Data snooping: what happens after testing two hundred ideas
Posted: Sat Oct 31, 2026 6:57 pm
Test enough ideas on the same data and some will look great by luck. A scalper who tries 200 variations of an entry rule will find several with impressive results, even if none has a real edge.
The statistical concept is multiple comparisons. A result that is significant at the 5 percent level will appear in about one of twenty random tests. After many tests, the best looks significant but is merely the luckiest.
Defenses include recording every test, including the failures, so that the number of trials is known. Adjust the required evidence accordingly: with more trials, demand stronger results. Use separate data for final validation that was never touched during development.
Another approach is requiring an economic reason for the edge. A rule with a plausible mechanism, such as a known liquidity pattern, deserves more trust than one that happened to work.
Out-of-sample forward testing remains the strongest check.
How many variations were tested before the current strategy was selected, and was that count tracked?
The statistical concept is multiple comparisons. A result that is significant at the 5 percent level will appear in about one of twenty random tests. After many tests, the best looks significant but is merely the luckiest.
Defenses include recording every test, including the failures, so that the number of trials is known. Adjust the required evidence accordingly: with more trials, demand stronger results. Use separate data for final validation that was never touched during development.
Another approach is requiring an economic reason for the edge. A rule with a plausible mechanism, such as a known liquidity pattern, deserves more trust than one that happened to work.
Out-of-sample forward testing remains the strongest check.
How many variations were tested before the current strategy was selected, and was that count tracked?