Why the test period chosen decides half the answer
Posted: Sun Nov 01, 2026 12:22 am
A strategy shows excellent numbers over a six-month test. The same logic over two years looks mediocre. The six months weren't picked dishonestly. They just happened to suit it.
Market regimes change. A trend-following strategy shines in trending phases and bleeds in ranges. A mean-reversion approach does the opposite. Volatility levels vary by year, spreads change, and the behavior of liquidity providers shifts.
For short-term strategies, regime changes occur at multiple scales: yearly, monthly, even daily. A strategy might work in the first hour of London and fail in the last hour of New York.
A solid test splits data into segments and looks at each. If profits come from one stretch only, the strategy is a bet on that regime continuing. If results are spread across periods, there's more reason for confidence.
Also worth checking is the effect of major events. Removing a single month, such as the aftermath of a central bank shock, sometimes eliminates most of the profit.
Looking at performance by month, by week, and by hour gives a far better picture than a single total.
A proper review divides the history into regimes. A trending period, a ranging period, a high-volatility period, a low-volatility one. Then it checks the strategy in each. A system that is positive across most of them is more likely to carry forward. One that depends on a single type is a market-timing bet.
Another useful test is to remove the best month and the worst month and look at what remains. If the strategy only worked because of one lucky stretch, it shows.
Years matter too. A strategy tested only on years with low spreads and strong trends might be unprepared for a market with higher costs and mean-reverting behavior.
The simple summary table, results by year, by quarter, and by hour, is not exciting to make, but it does more for confidence than any single headline number of total profit.
Which period in the data contributed the most to the final result, and would the strategy have survived without it?
Market regimes change. A trend-following strategy shines in trending phases and bleeds in ranges. A mean-reversion approach does the opposite. Volatility levels vary by year, spreads change, and the behavior of liquidity providers shifts.
For short-term strategies, regime changes occur at multiple scales: yearly, monthly, even daily. A strategy might work in the first hour of London and fail in the last hour of New York.
A solid test splits data into segments and looks at each. If profits come from one stretch only, the strategy is a bet on that regime continuing. If results are spread across periods, there's more reason for confidence.
Also worth checking is the effect of major events. Removing a single month, such as the aftermath of a central bank shock, sometimes eliminates most of the profit.
Looking at performance by month, by week, and by hour gives a far better picture than a single total.
A proper review divides the history into regimes. A trending period, a ranging period, a high-volatility period, a low-volatility one. Then it checks the strategy in each. A system that is positive across most of them is more likely to carry forward. One that depends on a single type is a market-timing bet.
Another useful test is to remove the best month and the worst month and look at what remains. If the strategy only worked because of one lucky stretch, it shows.
Years matter too. A strategy tested only on years with low spreads and strong trends might be unprepared for a market with higher costs and mean-reverting behavior.
The simple summary table, results by year, by quarter, and by hour, is not exciting to make, but it does more for confidence than any single headline number of total profit.
Which period in the data contributed the most to the final result, and would the strategy have survived without it?