Test Enough Strategies and One Will Look Great

[Fairman]

Test Enough Strategies and One Will Look Great

Scenario: a researcher tests 20 unrelated strategies, each with no real edge. They use the common standard that a result is "significant" if it would occur by chance less than 5 percent of the time. How likely is at least one false discovery?

Compute. The chance that a single no-edge strategy passes the test by luck is 0.05. The chance that it fails is 0.95. The chance that all 20 fail is 0.95 to the 20th power, which is about 0.358. So the chance of at least one false positive is 1 minus 0.358, about 64 percent.

With 50 strategies, 0.95 to the 50th is 0.077, so the chance of at least one false positive is about 92 percent. With 100 strategies, almost certain.

The same logic applies to indicators, parameter values, pair and timeframe combinations, and sessions. Every extra variation tested is another lottery ticket.

What do you do about it? Count how many things you tried. Apply a stricter threshold when the count is high, for example 0.05 divided by 20, which is 0.0025, a simple correction known as the Bonferroni adjustment. Require out-of-sample confirmation. And prefer ideas with a clear reason to work.

Look at how this plays out in marketing. A vendor who tested hundreds of settings and shows only the best is presenting the winner of a lottery.

Your own research process deserves the same discipline: write down every variant tested, and report the count alongside the best result.

Exercise: count the variations you have tried for your current strategy, and compute the chance that at least one looked good by luck.