A strategy has 14 parameters. After testing thousands of combinations, one set produces a beautiful equity curve. It feels like discovery.
More likely, the optimisation found the noise in the data and memorised it. The parameters fit the past precisely, and the future does not resemble the past precisely.
Signs of overfitting include extreme sensitivity, where a small change in one parameter wrecks the results, a very large number of rules for a small number of trades and performance that varies wildly between years.
Defences exist. Keep the rule set simple. Use out-of-sample data, reserving 30 percent of history that the optimisation never saw. Check that nearby parameter values also produce reasonable results, forming a plateau instead of a spike.
A robust strategy tends to look slightly mediocre in testing but holds up in live conditions. A stunning backtest is often a warning.
How many parameters does the main strategy use, and was it tested on unseen data?
Overfitting looks like skill until the market changes
-
Shadow Trader
- Posts: 5594
- Joined: Wed Sep 09, 2026 5:19 pm
-
FXS HongKong Scalper
- Posts: 548
- Joined: Sat Oct 24, 2026 10:15 pm
Re: Overfitting looks like skill until the market changes
Fourteen parameters is the giveaway. A rough rule that helps: if a parameter's best value changes a lot when you shift the test window by a few months, it is fitting noise. Rerun the optimisation on the first half and the second half of the data separately and compare the winning sets. When they barely overlap, the beautiful curve was memory, not discovery.
FXS Official Team | Hong Kong Session Desk
-
Amanhasnoname
- Posts: 2060
- Joined: Fri Oct 30, 2026 9:45 am
Re: Overfitting looks like skill until the market changes
My main scalp uses four parameters: entry offset from the M5 open, stop distance, target multiple, and a session start minute. I used to run eleven, including two moving averages and a volatility filter that changed best values every time I shifted the window by a quarter. That was the red flag.
I split history into two halves and re-optimised each separately. The winning sets barely overlapped on the filter lengths and the MA periods, while the four core numbers stayed inside a small band. I dropped the fragile knobs and kept the plateau. Out-of-sample on the reserved last thirty percent the simplified version lost a little polish but stayed positive; the fourteen-style monster collapsed.
A stunning backtest still makes me suspicious. If nearby values do not also work, I treat the peak as memory. Live results for the four-parameter version have been mediocre-to-decent for nine months, which is exactly what I want after watching prettier curves die.
I split history into two halves and re-optimised each separately. The winning sets barely overlapped on the filter lengths and the MA periods, while the four core numbers stayed inside a small band. I dropped the fragile knobs and kept the plateau. Out-of-sample on the reserved last thirty percent the simplified version lost a little polish but stayed positive; the fourteen-style monster collapsed.
A stunning backtest still makes me suspicious. If nearby values do not also work, I treat the peak as memory. Live results for the four-parameter version have been mediocre-to-decent for nine months, which is exactly what I want after watching prettier curves die.