Count the Tests Behind a Result

[Fairman]

Count the Tests Behind a Result

A trader tunes six parameters of a strategy, trying five values for each. They run every combination on 80 trades and pick the best. The best set shows a profit factor of 2.4. Should you trust it?

Count the combinations: 5 to the power of 6 is 15,625. The trader tested more than fifteen thousand versions on the same 80 trades.

Think about what that implies. If you flip enough coins, some will land heads ten times in a row by luck. With 15,625 variants, the best one will look good even if none has any real edge. Its profit factor is the maximum of many noisy numbers, which is biased upward.

How much? A rough rule from statistics: the expected best of N random results grows with the logarithm of N. With 15,625 trials, the best is likely about four standard deviations above the average, a result that would be extremely unusual for a single test.

A second view: 80 trades and six free parameters means about 13 trades per parameter. That is a thin base. Many practitioners want at least 30 trades per parameter, and often far more.

What should the trader do? Reduce parameters, test fewer values, require a plateau in results, and validate on fresh data.

Report the count. A result presented alongside the number of attempts tells you more than one given alone.

Exercise: count how many variations you have tried for your current strategy.

[FXS HongKong Scalper]

This is the most important point people miss in optimization. With 80 trades, the standard error of a profit factor is already large, and picking the best of 15,625 almost guarantees a number like 2.4 from noise.

A rough check we use: run the same grid on random entries with the same stop and target. If the best random version also gives profit factor 2 or more, the result tells you nothing. And it's better to look at the neighbourhood of the best set. If small changes in each parameter drop the result to 1.1, that's overfitting; if the area around it is consistently around 1.5, there may be something real.

[Shadow Trader]

When a large order hits the market, it consumes the available liquidity at several price levels. Price jumps through them in an instant, and what's left behind is a thin book with few resting orders near the new price.

For a few seconds after, the market is fragile. A small order can move price further than it should, and spreads stay wide until providers refill their quotes. That's a bad moment to enter. The fill is poor, and the price is more likely to snap back as the book refills.

Waiting five to ten seconds after a sudden jump lets quotes settle and shows whether the move has any support behind it. If the move holds while spreads normalise, it was probably real. If it fades as the spread tightens, it was a hole.

Has a short wait after a big move ever improved the quality of an entry?