Count the Tests Behind a Result
A trader tunes six parameters of a strategy, trying five values for each. They run every combination on 80 trades and pick the best. The best set shows a profit factor of 2.4. Should you trust it?
Count the combinations: 5 to the power of 6 is 15,625. The trader tested more than fifteen thousand versions on the same 80 trades.
Think about what that implies. If you flip enough coins, some will land heads ten times in a row by luck. With 15,625 variants, the best one will look good even if none has any real edge. Its profit factor is the maximum of many noisy numbers, which is biased upward.
How much? A rough rule from statistics: the expected best of N random results grows with the logarithm of N. With 15,625 trials, the best is likely about four standard deviations above the average, a result that would be extremely unusual for a single test.
A second view: 80 trades and six free parameters means about 13 trades per parameter. That is a thin base. Many practitioners want at least 30 trades per parameter, and often far more.
What should the trader do? Reduce parameters, test fewer values, require a plateau in results, and validate on fresh data.
Report the count. A result presented alongside the number of attempts tells you more than one given alone.
Exercise: count how many variations you have tried for your current strategy.
Count the Tests Behind a Result
Count the Tests Behind a Result
It’s Fairman 
-
FXS HongKong Scalper
- Posts: 543
- Joined: Sat Oct 24, 2026 10:15 pm
Re: Count the Tests Behind a Result
This is the most important point people miss in optimization. With 80 trades, the standard error of a profit factor is already large, and picking the best of 15,625 almost guarantees a number like 2.4 from noise.
A rough check we use: run the same grid on random entries with the same stop and target. If the best random version also gives profit factor 2 or more, the result tells you nothing. And it's better to look at the neighbourhood of the best set. If small changes in each parameter drop the result to 1.1, that's overfitting; if the area around it is consistently around 1.5, there may be something real.
A rough check we use: run the same grid on random entries with the same stop and target. If the best random version also gives profit factor 2 or more, the result tells you nothing. And it's better to look at the neighbourhood of the best set. If small changes in each parameter drop the result to 1.1, that's overfitting; if the area around it is consistently around 1.5, there may be something real.
FXS Official Team | Hong Kong Session Desk
-
Shadow Trader
- Posts: 5594
- Joined: Wed Sep 09, 2026 5:19 pm
Re: Count the Tests Behind a Result
When a large order hits the market, it consumes the available liquidity at several price levels. Price jumps through them in an instant, and what's left behind is a thin book with few resting orders near the new price.
For a few seconds after, the market is fragile. A small order can move price further than it should, and spreads stay wide until providers refill their quotes. That's a bad moment to enter. The fill is poor, and the price is more likely to snap back as the book refills.
Waiting five to ten seconds after a sudden jump lets quotes settle and shows whether the move has any support behind it. If the move holds while spreads normalise, it was probably real. If it fades as the spread tightens, it was a hole.
Has a short wait after a big move ever improved the quality of an entry?
For a few seconds after, the market is fragile. A small order can move price further than it should, and spreads stay wide until providers refill their quotes. That's a bad moment to enter. The fill is poor, and the price is more likely to snap back as the book refills.
Waiting five to ten seconds after a sudden jump lets quotes settle and shows whether the move has any support behind it. If the move holds while spreads normalise, it was probably real. If it fades as the spread tightens, it was a hole.
Has a short wait after a big move ever improved the quality of an entry?
Re: Count the Tests Behind a Result
The 15,625 combinations figure is striking, and the four standard deviation estimate is roughly right. For a large number of independent random results, the expected maximum grows roughly with the square root of two times the natural logarithm of the number of trials. For 15,625, that's the square root of about 19.3, which is about 4.4. So the best of the bunch would be expected to sit four or more standard deviations above average purely by chance. In real tests, the combinations aren't fully independent, since nearby parameter values produce similar results, so the effect is somewhat smaller, but still large.
It’s Fairman 
-
Amanhasnoname
- Posts: 2060
- Joined: Fri Oct 30, 2026 9:45 am
Re: Count the Tests Behind a Result
I fell into this trap with my first EA, and it cost me a month. I optimised a moving average cross with a stop, a target and a time filter, around 3,000 combinations, and the best one showed a profit factor of 2.1 on the previous year. On a live demo it fell back to 1.0 within six weeks. I had only counted the winner, not the three thousand I discarded.
The neighbourhood check from the reply above is what finally helped me. I now print the results as a grid and look for a flat area, not a spike. If the best cell is surrounded by cells that fail, I throw it out. I also keep a text file where I add one line for every batch I run, so the number of attempts is on record before I look at any result.
The random entry test is the one I would add to my routine next, since it shows right away what noise alone can produce on that data.
The neighbourhood check from the reply above is what finally helped me. I now print the results as a grid and look for a flat area, not a spike. If the best cell is surrounded by cells that fail, I throw it out. I also keep a text file where I add one line for every batch I run, so the number of attempts is on record before I look at any result.
The random entry test is the one I would add to my routine next, since it shows right away what noise alone can produce on that data.
Re: Count the Tests Behind a Result
The trades per parameter guideline is a useful rule of thumb. With 80 trades and six parameters, each parameter is fitted using about 13 trades' worth of information, which is very little. The strategy can bend itself around the specific trades in the sample. With 30 or more trades per parameter, the fit is more constrained, and the result is more likely to reflect a real pattern. Reducing the number of parameters is often the most effective fix. A strategy with two well chosen parameters and 80 trades is much more trustworthy than one with six.
It’s Fairman 
Re: Count the Tests Behind a Result
Requiring a plateau, as the post suggests, is one of the best defences. If the best parameter set sits in a region where nearby values also perform well, the result is more likely to be real. If it's an isolated peak, with neighbouring values performing poorly, it's probably fitted to noise. Plotting the results across parameter values, or looking at a heat map for two parameters, shows the shape. A broad plateau is reassuring. A sharp spike surrounded by poor results is a warning that the strategy is unlikely to perform as well on new data.
It’s Fairman 
Re: Count the Tests Behind a Result
Reporting the count alongside the result is a habit everyone should adopt. A profit factor of 2.4 from one test is impressive. The same number as the best of 15,625 tests is almost meaningless. Keeping a research log that records every variation tried makes the count honest and helps resist the temptation to keep tweaking. Has anyone here counted how many variations they've tested for their current strategy? I suspect the number would surprise most of us, and it might change how much confidence we place in our backtest results.
Re: Count the Tests Behind a Result
Fairman, haven't actually sat down and counted mine, to be fair, probably wouldnt want to know the number. I'm more sceptical of the four-sigma maths than the overfitting point itself... in practice the parameters are never independent like coin flips, so the real inflation is usually alot less dramatic than the formula suggests, mind you still enough to fool you.
Either way the fix is the same, fewer knobs and a proper out of sample test beats any clever statistic about how fooled you probably are.
G.
Either way the fix is the same, fewer knobs and a proper out of sample test beats any clever statistic about how fooled you probably are.
G.