P-hacking in strategy research
Posted: Wed Oct 28, 2026 6:02 am
Run enough tests, and something will look significant. That's p-hacking in a sentence, and it's the main way traders fool themselves with data.
Suppose 100 random, edge-free strategies are tested, each at a 5% significance level. About five will appear to have an edge purely by chance. A researcher who tries a hundred indicator settings, then reports the best, has done exactly that.
The hidden forms are subtle. Trying different timeframes, different pairs, different start dates, different stop distances, different filters, each variation is another test. Without counting the tests, the apparent significance is meaningless.
Protection comes from honest accounting. Record every variation tried. Adjust significance thresholds for the number of tests, for example with a Bonferroni correction, or demand much stronger results when many variations were tried. Reserve out-of-sample data for a single final check. Look for effects that survive across pairs and periods, not just in the best one.
A research log listing each test makes the process transparent.
Pre-registration is the cleanest defence. Before testing, write down the exact rule, the data range, the number of variations to be tried and the pass criterion. Seal it with a timestamp. When the test finishes, the result either meets the criterion or not, with no room for creative reinterpretation.
It sounds formal, but a plain text file in a folder does the job. Over time, the habit changes how ideas are formed, since vague hunches have to be turned into precise statements before they can be tested.
How many variations were tested before settling on the current strategy, and how was that accounted for?
Suppose 100 random, edge-free strategies are tested, each at a 5% significance level. About five will appear to have an edge purely by chance. A researcher who tries a hundred indicator settings, then reports the best, has done exactly that.
The hidden forms are subtle. Trying different timeframes, different pairs, different start dates, different stop distances, different filters, each variation is another test. Without counting the tests, the apparent significance is meaningless.
Protection comes from honest accounting. Record every variation tried. Adjust significance thresholds for the number of tests, for example with a Bonferroni correction, or demand much stronger results when many variations were tried. Reserve out-of-sample data for a single final check. Look for effects that survive across pairs and periods, not just in the best one.
A research log listing each test makes the process transparent.
Pre-registration is the cleanest defence. Before testing, write down the exact rule, the data range, the number of variations to be tried and the pass criterion. Seal it with a timestamp. When the test finishes, the result either meets the criterion or not, with no room for creative reinterpretation.
It sounds formal, but a plain text file in a folder does the job. Over time, the habit changes how ideas are formed, since vague hunches have to be turned into precise statements before they can be tested.
How many variations were tested before settling on the current strategy, and how was that accounted for?