Run enough tests, and something will look significant. That's p-hacking in a sentence, and it's the main way traders fool themselves with data.
Suppose 100 random, edge-free strategies are tested, each at a 5% significance level. About five will appear to have an edge purely by chance. A researcher who tries a hundred indicator settings, then reports the best, has done exactly that.
The hidden forms are subtle. Trying different timeframes, different pairs, different start dates, different stop distances, different filters, each variation is another test. Without counting the tests, the apparent significance is meaningless.
Protection comes from honest accounting. Record every variation tried. Adjust significance thresholds for the number of tests, for example with a Bonferroni correction, or demand much stronger results when many variations were tried. Reserve out-of-sample data for a single final check. Look for effects that survive across pairs and periods, not just in the best one.
A research log listing each test makes the process transparent.
Pre-registration is the cleanest defence. Before testing, write down the exact rule, the data range, the number of variations to be tried and the pass criterion. Seal it with a timestamp. When the test finishes, the result either meets the criterion or not, with no room for creative reinterpretation.
It sounds formal, but a plain text file in a folder does the job. Over time, the habit changes how ideas are formed, since vague hunches have to be turned into precise statements before they can be tested.
How many variations were tested before settling on the current strategy, and how was that accounted for?
P-hacking in strategy research
-
FXS London Scalper
- Posts: 979
- Joined: Sun Oct 18, 2026 12:25 pm
Re: P-hacking in strategy research
Five false positives out of 100 random strategies is the arithmetic every optimisation report should start with. For London breakout research the trap is subtle: testing range windows of 30, 45, 60 and 90 minutes, three stop sizes and four targets is already 48 strategies. A held-out year, touched once, is the only check that survives that many tries.
FXS Official Team | London Session Desk
-
Amanhasnoname
- Posts: 2060
- Joined: Fri Oct 30, 2026 9:45 am
Re: P-hacking in strategy research
Before I locked my current London open fade I had already tried 63 variations without writing them down. Different range windows, three ATR multiples for stops, four targets, and two filters on overnight gaps. The best equity curve looked brilliant until I admitted the search size. At a five percent threshold that many tries almost guarantees a few false winners.
I started a plain text research log after that. Each idea gets a one-line rule, the date range, how many knobs I am allowed to turn, and a pass criterion sealed before I run anything. The first strategy that survived a single held-out year under those rules was far duller than the optimised curves, and it is the one still running.
Counting the tests changed the psychology more than any formula. Vague hunches now have to become precise statements before they touch data. That alone killed half the ideas that used to waste a weekend of curve fitting.
I started a plain text research log after that. Each idea gets a one-line rule, the date range, how many knobs I am allowed to turn, and a pass criterion sealed before I run anything. The first strategy that survived a single held-out year under those rules was far duller than the optimised curves, and it is the one still running.
Counting the tests changed the psychology more than any formula. Vague hunches now have to become precise statements before they touch data. That alone killed half the ideas that used to waste a weekend of curve fitting.