Advertisement IC Markets

P-hacking in strategy research

Discuss 1-minute to 15-minute price action setups, fading intraday momentum, key support/resistance zones, and proven short-term trading methodologies.
Post Reply
Shadow Trader
Posts: 5594
Joined: Wed Sep 09, 2026 5:19 pm

P-hacking in strategy research

Post by Shadow Trader »

Run enough tests, and something will look significant. That's p-hacking in a sentence, and it's the main way traders fool themselves with data.

Suppose 100 random, edge-free strategies are tested, each at a 5% significance level. About five will appear to have an edge purely by chance. A researcher who tries a hundred indicator settings, then reports the best, has done exactly that.

The hidden forms are subtle. Trying different timeframes, different pairs, different start dates, different stop distances, different filters, each variation is another test. Without counting the tests, the apparent significance is meaningless.

Protection comes from honest accounting. Record every variation tried. Adjust significance thresholds for the number of tests, for example with a Bonferroni correction, or demand much stronger results when many variations were tried. Reserve out-of-sample data for a single final check. Look for effects that survive across pairs and periods, not just in the best one.

A research log listing each test makes the process transparent.

Pre-registration is the cleanest defence. Before testing, write down the exact rule, the data range, the number of variations to be tried and the pass criterion. Seal it with a timestamp. When the test finishes, the result either meets the criterion or not, with no room for creative reinterpretation.

It sounds formal, but a plain text file in a folder does the job. Over time, the habit changes how ideas are formed, since vague hunches have to be turned into precise statements before they can be tested.

How many variations were tested before settling on the current strategy, and how was that accounted for?
Recommended broker for automated trading & scalping IC Markets
FXS London Scalper
Posts: 979
Joined: Sun Oct 18, 2026 12:25 pm

Re: P-hacking in strategy research

Post by FXS London Scalper »

Shadow Trader wrote: Reserve out-of-sample data for a single final check.
Five false positives out of 100 random strategies is the arithmetic every optimisation report should start with. For London breakout research the trap is subtle: testing range windows of 30, 45, 60 and 90 minutes, three stop sizes and four targets is already 48 strategies. A held-out year, touched once, is the only check that survives that many tries.
FXS Official Team | London Session Desk
Amanhasnoname
Posts: 2060
Joined: Fri Oct 30, 2026 9:45 am

Re: P-hacking in strategy research

Post by Amanhasnoname »

Before I locked my current London open fade I had already tried 63 variations without writing them down. Different range windows, three ATR multiples for stops, four targets, and two filters on overnight gaps. The best equity curve looked brilliant until I admitted the search size. At a five percent threshold that many tries almost guarantees a few false winners.

I started a plain text research log after that. Each idea gets a one-line rule, the date range, how many knobs I am allowed to turn, and a pass criterion sealed before I run anything. The first strategy that survived a single held-out year under those rules was far duller than the optimised curves, and it is the one still running.

Counting the tests changed the psychology more than any formula. Vague hunches now have to become precise statements before they touch data. That alone killed half the ideas that used to waste a weekend of curve fitting.
Post Reply