Overfitting: When a Strategy Is Built to Explain the Past

[Fairman]

Overfitting: When a Strategy Is Built to Explain the Past

Overfitting happens when a strategy is tailored so closely to historical data that it captures noise instead of real patterns. It looks excellent in testing and fails in live trading.

How does it occur? You test an idea, see losses, and add a filter. You add another rule for a specific bad period. You adjust the settings until the equity curve smooths out. Each tweak seems reasonable, but together they carve a strategy around the quirks of the sample.

The more parameters and rules you have relative to the number of trades, the greater the risk. A strategy with ten adjustable settings tested on 80 trades has essentially memorized them.

Signs of overfitting include sensitivity to small parameter changes, rules that lack a plausible explanation, an unusually smooth equity curve, and sharp deterioration on new data.

Defenses are straightforward. Keep strategies simple with few parameters. Prefer rules with a logical reason. Test on multiple pairs and time periods. Hold back part of the data that you never look at while designing, and test on it at the end.

Expect live results to be worse than backtested results. A common rule of thumb is to assume a meaningful haircut.

If a result looks too good, it probably is.

Practical step: count the number of adjustable rules and parameters in your strategy, and compare with your number of tested trades.

[Fairman]

Easy check for this: nudge every setting 10 to 20 percent each way and rerun. A real idea gets a bit worse. An overfit one falls off a cliff.

[Fairman]

The trap for me was always "just one more filter" after a bad month. Now any new rule has to make sense on its own before I look at whether it improves the numbers, not after.

[Shadow Trader]

The nudge test is the one I run first, but it needs a different step size for settings that are not smooth numbers. Moving a session start of 08:00 by 15 percent means nothing, so for times I shift by 15 minutes each way and for pairs I swap in the nearest cousin, GBPUSD for EURUSD.

The holdout has a quieter trap. It works once, and the moment you see it fail, tweak, and rerun, it has become part of the design sample.

I keep a short log of every time I opened the holdout. After the second look I treat it as burned and wait for fresh months of data instead of pretending it is still clean.

On counting rules against trades, my rough line is about 30 trades per adjustable setting before I believe anything. Ten settings on 80 trades fails that easily, and so did my own breakout system with six settings and 140 trades.

Fairman, how do you treat a filter that only removes a handful of trades? Those always look harmless to me, and they are the ones I end up regretting.