A strategy shows excellent numbers over a six-month test. The same logic over two years looks mediocre. The six months weren't picked dishonestly. They just happened to suit it.
Market regimes change. A trend-following strategy shines in trending phases and bleeds in ranges. A mean-reversion approach does the opposite. Volatility levels vary by year, spreads change, and the behavior of liquidity providers shifts.
For short-term strategies, regime changes occur at multiple scales: yearly, monthly, even daily. A strategy might work in the first hour of London and fail in the last hour of New York.
A solid test splits data into segments and looks at each. If profits come from one stretch only, the strategy is a bet on that regime continuing. If results are spread across periods, there's more reason for confidence.
Also worth checking is the effect of major events. Removing a single month, such as the aftermath of a central bank shock, sometimes eliminates most of the profit.
Looking at performance by month, by week, and by hour gives a far better picture than a single total.
A proper review divides the history into regimes. A trending period, a ranging period, a high-volatility period, a low-volatility one. Then it checks the strategy in each. A system that is positive across most of them is more likely to carry forward. One that depends on a single type is a market-timing bet.
Another useful test is to remove the best month and the worst month and look at what remains. If the strategy only worked because of one lucky stretch, it shows.
Years matter too. A strategy tested only on years with low spreads and strong trends might be unprepared for a market with higher costs and mean-reverting behavior.
The simple summary table, results by year, by quarter, and by hour, is not exciting to make, but it does more for confidence than any single headline number of total profit.
Which period in the data contributed the most to the final result, and would the strategy have survived without it?
Why the test period chosen decides half the answer
-
Shadow Trader
- Posts: 5594
- Joined: Wed Sep 09, 2026 5:19 pm
-
FXS London Scalper
- Posts: 978
- Joined: Sun Oct 18, 2026 12:25 pm
Re: Why the test period chosen decides half the answer
The regime point is why a test should always include at least one trending and one ranging stretch. For London breakout systems, summer months often look very different from autumn, so a six-month window in either season gives a misleading picture.
FXS Official Team | London Session Desk
-
Amanhasnoname
- Posts: 2060
- Joined: Fri Oct 30, 2026 9:45 am
Re: Why the test period chosen decides half the answer
When I backtested my Frankfurt open fade a couple of years ago, the first run covered March to August and the equity curve looked almost too smooth. Out of stubbornness I extended it back to the start of the previous year and the whole thing went flat. Nearly all of the profit sat in a ten week stretch where EURUSD kept reversing its opening spike.
What I do now is break every test into calendar quarters and also by the first, second and third hour after the open. I put the results into a simple grid and look for empty cells. If more than one quarter in four is negative, or one hour carries the rest, I treat the idea as unfinished rather than broken.
The other habit is keeping a short note beside each quarter describing what the market was doing, like steady trend, choppy range or post-meeting volatility. It takes ten minutes and it usually explains the weak periods better than any extra statistic does.
What I do now is break every test into calendar quarters and also by the first, second and third hour after the open. I put the results into a simple grid and look for empty cells. If more than one quarter in four is negative, or one hour carries the rest, I treat the idea as unfinished rather than broken.
The other habit is keeping a short note beside each quarter describing what the market was doing, like steady trend, choppy range or post-meeting volatility. It takes ten minutes and it usually explains the weak periods better than any extra statistic does.