Optimizing Parameters Without Fooling Yourself
Optimization means testing different settings to find the best-performing ones. Done carefully, it improves a strategy. Done carelessly, it creates fantasies.
The key is to look for plateaus, not peaks. If a moving average period of 20 gives the best result, check 15, 18, 22, and 25. If they all perform reasonably, the idea is robust, and the best number matters little. If only 20 works, and neighbors fail, you have found noise.
Limit the number of parameters you optimize. Each additional parameter multiplies the combinations and the chance of finding something that fits by luck. Optimizing two settings is manageable. Optimizing eight is a recipe for overfitting.
Use a sensible performance measure. Maximizing net profit alone often selects risky settings. Consider metrics that include drawdown, such as profit divided by maximum drawdown, or expectancy in R with a minimum number of trades.
Choose the middle of a stable region instead of the single best point. It tends to hold up better in the future.
Keep part of the data untouched for validation, and run the chosen settings on it once.
Finally, accept that optimal settings drift over time, and periodic review is better than constant retuning.
Practical step: pick one parameter in your strategy and produce a table of results for five values on each side of your current choice.
Optimizing Parameters Without Fooling Yourself
Optimizing Parameters Without Fooling Yourself
It’s Fairman 
Re: Optimizing Parameters Without Fooling Yourself
A heatmap helps a lot if your platform can do one with two parameters. You want a broad warm area, not one hot square surrounded by cold.
It’s Fairman 
Re: Optimizing Parameters Without Fooling Yourself
I'd also say set your minimum trade count before you start optimizing. Otherwise the best result is often the setting that only took 15 trades and happened to win most of them.
It’s Fairman 
-
Shadow Trader
- Posts: 5594
- Joined: Wed Sep 09, 2026 5:19 pm
Re: Optimizing Parameters Without Fooling Yourself
One way to make the plateau idea concrete is to sweep a single setting and put the results side by side. Say you test a moving average filter from period 10 to 30 on the same data for two versions of a setup.
In the chart, version A peaks at 48R at period 20, but period 18 gives only 9R and period 22 gives 6R. Version B never gets above 31R, yet everything from 16 to 26 sits between 24R and 31R.
Most people would pick A from a results table sorted by profit. B is the one worth trading, and I would take a value near 21 from the middle of its flat stretch rather than its single best point.
The arithmetic on parameter counts deserves spelling out as well. Eight settings with ten values each is 100 million combinations, so something will look brilliant by chance alone.
For the hold out data, decide in advance what counts as a pass. Something like at least half the in sample R per trade, with drawdown no worse than 1.5 times the test figure, stops you rationalising a weak result afterwards.
In the chart, version A peaks at 48R at period 20, but period 18 gives only 9R and period 22 gives 6R. Version B never gets above 31R, yet everything from 16 to 26 sits between 24R and 31R.
Most people would pick A from a results table sorted by profit. B is the one worth trading, and I would take a value near 21 from the middle of its flat stretch rather than its single best point.
The arithmetic on parameter counts deserves spelling out as well. Eight settings with ten values each is 100 million combinations, so something will look brilliant by chance alone.
For the hold out data, decide in advance what counts as a pass. Something like at least half the in sample R per trade, with drawdown no worse than 1.5 times the test figure, stops you rationalising a weak result afterwards.
-
FXS Prop Desk
- Posts: 1689
- Joined: Wed Oct 07, 2026 9:52 am
Re: Optimizing Parameters Without Fooling Yourself
The plateau test is an important point. One way to visualise it is a heatmap of results for two parameters at a time. A strategy with a broad area of acceptable results is much more robust than one with a single bright spot. Many platforms, including MT5's optimizer, can export the results in a format that makes this easy.
Re-running the test on a different period confirms the plateau.
Re-running the test on a different period confirms the plateau.
FXS Official Team | The Idea Lab & Strategy
-
Shadow Trader
- Posts: 5594
- Joined: Wed Sep 09, 2026 5:19 pm
Re: Optimizing Parameters Without Fooling Yourself
Re-running on a different period, as FXS Prop Desk suggested, can be pushed one step further into a walk forward test. Optimise on 12 months, trade the chosen setting untouched on the next 3, then slide the whole window forward and repeat.
Six rounds of that give six out of sample results instead of one. A single hold out period can pass or fail mostly on luck, which makes the pass rule from my earlier post, keeping half or more of the in sample result, harder to apply fairly.
The chart (illustration) shows a made up run of six windows. In sample results stay close to 0.30R per trade every time, while the out of sample figures range from 0.19R down to minus 0.04R and average 0.12R.
That average is 40 percent of the in sample figure, short of the half rule even though five of the six windows made money. Window 3 on its own would have passed comfortably and window 4 on its own would have failed, which is exactly the luck problem.
A ratio that low usually means the optimiser is finding part edge and part noise. Widening the plateau, dropping a parameter, or accepting a smaller expected R are all more honest than retuning until the number looks better.
Six rounds of that give six out of sample results instead of one. A single hold out period can pass or fail mostly on luck, which makes the pass rule from my earlier post, keeping half or more of the in sample result, harder to apply fairly.
The chart (illustration) shows a made up run of six windows. In sample results stay close to 0.30R per trade every time, while the out of sample figures range from 0.19R down to minus 0.04R and average 0.12R.
That average is 40 percent of the in sample figure, short of the half rule even though five of the six windows made money. Window 3 on its own would have passed comfortably and window 4 on its own would have failed, which is exactly the luck problem.
A ratio that low usually means the optimiser is finding part edge and part noise. Widening the plateau, dropping a parameter, or accepting a smaller expected R are all more honest than retuning until the number looks better.