Survivorship bias in pair selection: testing only what worked
Posted: Sun Oct 25, 2026 10:03 am
A trader backtests a sweep strategy on five pairs and finds it works well on two. They decide to trade only those two, and report impressive results. The danger is that the selection itself is a form of data mining. With enough pairs and settings, some will look good by chance. If the strategy has no real edge, you will still find a few combinations that appear profitable over a given period. To protect against this, I use a few safeguards. I define the pairs in advance based on liquidity and spread, not on past performance. I test on one period and then validate on a different period I have not looked at. And I look for a logical reason why the strategy should work on a pair, such as session overlap or volatility characteristics. If a pair works only in one specific year and fails in the others, I treat it as noise. A strategy with a genuine edge should show at least modest results across most reasonable instruments and periods, even if not equally strong everywhere.