Advertisement IC Markets

Overfitting: When a Strategy Is Built to Explain the Past

Share, develop, and backtest custom MQL4/MQL5 Expert Advisors, Python data-scraping scripts, trading bots, and automated market alert systems.
Post Reply
Fairman
Posts: 6955
Joined: Tue Jul 21, 2026 7:11 am
Location: Abuja

Overfitting: When a Strategy Is Built to Explain the Past

Post by Fairman »

Overfitting: When a Strategy Is Built to Explain the Past

Overfitting happens when a strategy is tailored so closely to historical data that it captures noise instead of real patterns. It looks excellent in testing and fails in live trading.

How does it occur? You test an idea, see losses, and add a filter. You add another rule for a specific bad period. You adjust the settings until the equity curve smooths out. Each tweak seems reasonable, but together they carve a strategy around the quirks of the sample.

The more parameters and rules you have relative to the number of trades, the greater the risk. A strategy with ten adjustable settings tested on 80 trades has essentially memorized them.

Signs of overfitting include sensitivity to small parameter changes, rules that lack a plausible explanation, an unusually smooth equity curve, and sharp deterioration on new data.

Defenses are straightforward. Keep strategies simple with few parameters. Prefer rules with a logical reason. Test on multiple pairs and time periods. Hold back part of the data that you never look at while designing, and test on it at the end.

Expect live results to be worse than backtested results. A common rule of thumb is to assume a meaningful haircut.

If a result looks too good, it probably is.

Practical step: count the number of adjustable rules and parameters in your strategy, and compare with your number of tested trades.
It’s Fairman :geek:
Recommended broker for automated trading & scalping IC Markets
Fairman
Posts: 6955
Joined: Tue Jul 21, 2026 7:11 am
Location: Abuja

Re: Overfitting: When a Strategy Is Built to Explain the Past

Post by Fairman »

Easy check for this: nudge every setting 10 to 20 percent each way and rerun. A real idea gets a bit worse. An overfit one falls off a cliff.
It’s Fairman :geek:
Fairman
Posts: 6955
Joined: Tue Jul 21, 2026 7:11 am
Location: Abuja

Re: Overfitting: When a Strategy Is Built to Explain the Past

Post by Fairman »

The trap for me was always "just one more filter" after a bad month. Now any new rule has to make sense on its own before I look at whether it improves the numbers, not after.
It’s Fairman :geek:
Shadow Trader
Posts: 5594
Joined: Wed Sep 09, 2026 5:19 pm

Re: Overfitting: When a Strategy Is Built to Explain the Past

Post by Shadow Trader »

The nudge test is the one I run first, but it needs a different step size for settings that are not smooth numbers. Moving a session start of 08:00 by 15 percent means nothing, so for times I shift by 15 minutes each way and for pairs I swap in the nearest cousin, GBPUSD for EURUSD.

The holdout has a quieter trap. It works once, and the moment you see it fail, tweak, and rerun, it has become part of the design sample.

I keep a short log of every time I opened the holdout. After the second look I treat it as burned and wait for fresh months of data instead of pretending it is still clean.

On counting rules against trades, my rough line is about 30 trades per adjustable setting before I believe anything. Ten settings on 80 trades fails that easily, and so did my own breakout system with six settings and 140 trades.

Fairman, how do you treat a filter that only removes a handful of trades? Those always look harmless to me, and they are the ones I end up regretting.
Shadow Trader
Posts: 5594
Joined: Wed Sep 09, 2026 5:19 pm

Re: Overfitting: When a Strategy Is Built to Explain the Past

Post by Shadow Trader »

Fairman's "just one more filter" trap has a sibling that is harder to see, which is how many versions get tried before one is kept. Every discarded variant still counts, because the survivor was picked for looking good on the very data used to judge it.
img_12989_r2.png
img_12989_r2.png (52.05 KiB) Viewed 236 times
Imagine 40 variants of a rule with no edge at all, each producing 150 trades that are pure coin flips at 1R either way. The chart shows a made up run of that, where the best variant finishes at plus 26R and 10 of the 40 clear plus 10R.

Nobody would keep the losers, so the plus 26R version is the one that ends up in a forum thread with a smooth curve. Its expected result on the next 150 trades is still zero.

The defense I use is to write down the number of variants tried next to the result. If forty were tested, the winner has to beat what the best of forty random ones would show, not just beat zero.

That is also why the holdout matters more as the variant count grows. A clean result on unseen months is one of the few things the selection step cannot fake.
FXS Prop Desk
Posts: 1689
Joined: Wed Oct 07, 2026 9:52 am

Re: Overfitting: When a Strategy Is Built to Explain the Past

Post by FXS Prop Desk »

A useful check for overfitting is to count the parameters and rules relative to the number of trades. A strategy with ten conditions tested on eighty trades is almost certainly fitted to noise. Another is to perturb each parameter slightly: if a moving average of 21 works but 19 and 23 fail, the result is fragile. Robust strategies show a broad plateau of acceptable results rather than a sharp peak.
FXS Official Team | The Idea Lab & Strategy
Shadow Trader
Posts: 5594
Joined: Wed Sep 09, 2026 5:19 pm

Re: Overfitting: When a Strategy Is Built to Explain the Past

Post by Shadow Trader »

Adding filters to a strategy often improves its backtest, and each addition seems logical. Over time, though, the stack of conditions can make the strategy so specific that it fits history rather than reflecting a genuine edge.

Each filter reduces the number of trades, so the final sample becomes small. A system with six filters may trade only a handful of times per month, and its historical results rest on a thin base. Random luck can account for much of the apparent performance.

A disciplined approach requires evidence for each filter. Does it improve out-of-sample results, and is the improvement large relative to its uncertainty? Does it have a plausible reason to work? Filters that survive both tests earn a place. Testing the final system with each filter removed one at a time shows how much each contributes. Simpler strategies, with fewer rules and more trades, usually prove more robust when conditions change.
Post Reply