Overfitting: When a Strategy Is Built to Explain the Past
Overfitting happens when a strategy is tailored so closely to historical data that it captures noise instead of real patterns. It looks excellent in testing and fails in live trading.
How does it occur? You test an idea, see losses, and add a filter. You add another rule for a specific bad period. You adjust the settings until the equity curve smooths out. Each tweak seems reasonable, but together they carve a strategy around the quirks of the sample.
The more parameters and rules you have relative to the number of trades, the greater the risk. A strategy with ten adjustable settings tested on 80 trades has essentially memorized them.
Signs of overfitting include sensitivity to small parameter changes, rules that lack a plausible explanation, an unusually smooth equity curve, and sharp deterioration on new data.
Defenses are straightforward. Keep strategies simple with few parameters. Prefer rules with a logical reason. Test on multiple pairs and time periods. Hold back part of the data that you never look at while designing, and test on it at the end.
Expect live results to be worse than backtested results. A common rule of thumb is to assume a meaningful haircut.
If a result looks too good, it probably is.
Practical step: count the number of adjustable rules and parameters in your strategy, and compare with your number of tested trades.
Overfitting: When a Strategy Is Built to Explain the Past
Re: Overfitting: When a Strategy Is Built to Explain the Past
Easy check for this: nudge every setting 10 to 20 percent each way and rerun. A real idea gets a bit worse. An overfit one falls off a cliff.
It’s Fairman 
Re: Overfitting: When a Strategy Is Built to Explain the Past
The trap for me was always "just one more filter" after a bad month. Now any new rule has to make sense on its own before I look at whether it improves the numbers, not after.
It’s Fairman 
-
Shadow Trader
- Posts: 5594
- Joined: Wed Sep 09, 2026 5:19 pm
Re: Overfitting: When a Strategy Is Built to Explain the Past
The nudge test is the one I run first, but it needs a different step size for settings that are not smooth numbers. Moving a session start of 08:00 by 15 percent means nothing, so for times I shift by 15 minutes each way and for pairs I swap in the nearest cousin, GBPUSD for EURUSD.
The holdout has a quieter trap. It works once, and the moment you see it fail, tweak, and rerun, it has become part of the design sample.
I keep a short log of every time I opened the holdout. After the second look I treat it as burned and wait for fresh months of data instead of pretending it is still clean.
On counting rules against trades, my rough line is about 30 trades per adjustable setting before I believe anything. Ten settings on 80 trades fails that easily, and so did my own breakout system with six settings and 140 trades.
Fairman, how do you treat a filter that only removes a handful of trades? Those always look harmless to me, and they are the ones I end up regretting.
The holdout has a quieter trap. It works once, and the moment you see it fail, tweak, and rerun, it has become part of the design sample.
I keep a short log of every time I opened the holdout. After the second look I treat it as burned and wait for fresh months of data instead of pretending it is still clean.
On counting rules against trades, my rough line is about 30 trades per adjustable setting before I believe anything. Ten settings on 80 trades fails that easily, and so did my own breakout system with six settings and 140 trades.
Fairman, how do you treat a filter that only removes a handful of trades? Those always look harmless to me, and they are the ones I end up regretting.
-
Shadow Trader
- Posts: 5594
- Joined: Wed Sep 09, 2026 5:19 pm
Re: Overfitting: When a Strategy Is Built to Explain the Past
Fairman's "just one more filter" trap has a sibling that is harder to see, which is how many versions get tried before one is kept. Every discarded variant still counts, because the survivor was picked for looking good on the very data used to judge it.
Imagine 40 variants of a rule with no edge at all, each producing 150 trades that are pure coin flips at 1R either way. The chart shows a made up run of that, where the best variant finishes at plus 26R and 10 of the 40 clear plus 10R.
Nobody would keep the losers, so the plus 26R version is the one that ends up in a forum thread with a smooth curve. Its expected result on the next 150 trades is still zero.
The defense I use is to write down the number of variants tried next to the result. If forty were tested, the winner has to beat what the best of forty random ones would show, not just beat zero.
That is also why the holdout matters more as the variant count grows. A clean result on unseen months is one of the few things the selection step cannot fake.
Imagine 40 variants of a rule with no edge at all, each producing 150 trades that are pure coin flips at 1R either way. The chart shows a made up run of that, where the best variant finishes at plus 26R and 10 of the 40 clear plus 10R.
Nobody would keep the losers, so the plus 26R version is the one that ends up in a forum thread with a smooth curve. Its expected result on the next 150 trades is still zero.
The defense I use is to write down the number of variants tried next to the result. If forty were tested, the winner has to beat what the best of forty random ones would show, not just beat zero.
That is also why the holdout matters more as the variant count grows. A clean result on unseen months is one of the few things the selection step cannot fake.
-
FXS Prop Desk
- Posts: 1689
- Joined: Wed Oct 07, 2026 9:52 am
Re: Overfitting: When a Strategy Is Built to Explain the Past
A useful check for overfitting is to count the parameters and rules relative to the number of trades. A strategy with ten conditions tested on eighty trades is almost certainly fitted to noise. Another is to perturb each parameter slightly: if a moving average of 21 works but 19 and 23 fail, the result is fragile. Robust strategies show a broad plateau of acceptable results rather than a sharp peak.
FXS Official Team | The Idea Lab & Strategy
-
Shadow Trader
- Posts: 5594
- Joined: Wed Sep 09, 2026 5:19 pm
Re: Overfitting: When a Strategy Is Built to Explain the Past
Adding filters to a strategy often improves its backtest, and each addition seems logical. Over time, though, the stack of conditions can make the strategy so specific that it fits history rather than reflecting a genuine edge.
Each filter reduces the number of trades, so the final sample becomes small. A system with six filters may trade only a handful of times per month, and its historical results rest on a thin base. Random luck can account for much of the apparent performance.
A disciplined approach requires evidence for each filter. Does it improve out-of-sample results, and is the improvement large relative to its uncertainty? Does it have a plausible reason to work? Filters that survive both tests earn a place. Testing the final system with each filter removed one at a time shows how much each contributes. Simpler strategies, with fewer rules and more trades, usually prove more robust when conditions change.
Each filter reduces the number of trades, so the final sample becomes small. A system with six filters may trade only a handful of times per month, and its historical results rest on a thin base. Random luck can account for much of the apparent performance.
A disciplined approach requires evidence for each filter. Does it improve out-of-sample results, and is the improvement large relative to its uncertainty? Does it have a plausible reason to work? Filters that survive both tests earn a place. Testing the final system with each filter removed one at a time shows how much each contributes. Simpler strategies, with fewer rules and more trades, usually prove more robust when conditions change.