Count the Tests Behind a Result

[Fairman]

Count the Tests Behind a Result

A trader tunes six parameters of a strategy, trying five values for each. They run every combination on 80 trades and pick the best. The best set shows a profit factor of 2.4. Should you trust it?

Count the combinations: 5 to the power of 6 is 15,625. The trader tested more than fifteen thousand versions on the same 80 trades.

Think about what that implies. If you flip enough coins, some will land heads ten times in a row by luck. With 15,625 variants, the best one will look good even if none has any real edge. Its profit factor is the maximum of many noisy numbers, which is biased upward.

How much? A rough rule from statistics: the expected best of N random results grows with the logarithm of N. With 15,625 trials, the best is likely about four standard deviations above the average, a result that would be extremely unusual for a single test.

A second view: 80 trades and six free parameters means about 13 trades per parameter. That is a thin base. Many practitioners want at least 30 trades per parameter, and often far more.

What should the trader do? Reduce parameters, test fewer values, require a plateau in results, and validate on fresh data.

Report the count. A result presented alongside the number of attempts tells you more than one given alone.

Exercise: count how many variations you have tried for your current strategy.

[FXS HongKong Scalper]

This is the most important point people miss in optimization. With 80 trades, the standard error of a profit factor is already large, and picking the best of 15,625 almost guarantees a number like 2.4 from noise.

A rough check we use: run the same grid on random entries with the same stop and target. If the best random version also gives profit factor 2 or more, the result tells you nothing. And it's better to look at the neighbourhood of the best set. If small changes in each parameter drop the result to 1.1, that's overfitting; if the area around it is consistently around 1.5, there may be something real.