Test Enough Strategies and One Will Look Great
Scenario: a researcher tests 20 unrelated strategies, each with no real edge. They use the common standard that a result is "significant" if it would occur by chance less than 5 percent of the time. How likely is at least one false discovery?
Compute. The chance that a single no-edge strategy passes the test by luck is 0.05. The chance that it fails is 0.95. The chance that all 20 fail is 0.95 to the 20th power, which is about 0.358. So the chance of at least one false positive is 1 minus 0.358, about 64 percent.
With 50 strategies, 0.95 to the 50th is 0.077, so the chance of at least one false positive is about 92 percent. With 100 strategies, almost certain.
The same logic applies to indicators, parameter values, pair and timeframe combinations, and sessions. Every extra variation tested is another lottery ticket.
What do you do about it? Count how many things you tried. Apply a stricter threshold when the count is high, for example 0.05 divided by 20, which is 0.0025, a simple correction known as the Bonferroni adjustment. Require out-of-sample confirmation. And prefer ideas with a clear reason to work.
Look at how this plays out in marketing. A vendor who tested hundreds of settings and shows only the best is presenting the winner of a lottery.
Your own research process deserves the same discipline: write down every variant tested, and report the count alongside the best result.
Exercise: count the variations you have tried for your current strategy, and compute the chance that at least one looked good by luck.
Test Enough Strategies and One Will Look Great
Test Enough Strategies and One Will Look Great
It’s Fairman 
-
Shadow Trader
- Posts: 5594
- Joined: Wed Sep 09, 2026 5:19 pm
Re: Test Enough Strategies and One Will Look Great
Some sessions feel perfect. Decisions come without effort, entries land, exits are crisp. Time disappears. Athletes call it flow.
Scalpers experience it too, and it's rewarding. The danger lies in what happens at the edges. Inside flow, self-monitoring drops. The usual checks, such as the clock, the daily limit, the signs of fatigue, fade out of awareness.
Flow also ends without notice. The market changes character, or concentration slips, and the trader keeps acting as if the state continues. The trades that follow are taken with the confidence of flow but without its accuracy.
Another trap is chasing the state. After a great session, the next one starts with the goal of recreating the feeling, which pushes toward overtrading and forced entries.
Some use external anchors to stay grounded: a timer that sounds every 30 minutes, prompting a quick check of the trade count and mood. The interruption costs a little flow and saves a lot of damage.
Flow is a result of good conditions, not a switch to flip.
Did a great streak ever end in a run of avoidable losses, and was the shift spotted in time?
Scalpers experience it too, and it's rewarding. The danger lies in what happens at the edges. Inside flow, self-monitoring drops. The usual checks, such as the clock, the daily limit, the signs of fatigue, fade out of awareness.
Flow also ends without notice. The market changes character, or concentration slips, and the trader keeps acting as if the state continues. The trades that follow are taken with the confidence of flow but without its accuracy.
Another trap is chasing the state. After a great session, the next one starts with the goal of recreating the feeling, which pushes toward overtrading and forced entries.
Some use external anchors to stay grounded: a timer that sounds every 30 minutes, prompting a quick check of the trade count and mood. The interruption costs a little flow and saves a lot of damage.
Flow is a result of good conditions, not a switch to flip.
Did a great streak ever end in a run of avoidable losses, and was the shift spotted in time?