Survivorship in strategy testing
Posted: Wed Oct 28, 2026 6:08 am
Survivorship bias isn't only about brokers and forum posters. It infects strategy testing too.
When a researcher builds dozens of strategies, keeps the ones that work, and discards the rest, the surviving set looks impressive. The discarded failures are invisible, so the success rate of the process is overstated. Anyone reading the final report sees only the winners.
The same applies to data. Backtests on instruments that exist today ignore those that were delisted, or in forex, brokers or pairs that no longer trade. Indicators that survived years of use are those that happened to work, and the ones that were dropped aren't in the textbooks.
Defence starts with counting: how many ideas were tested, and how many survived? A one-in-twenty success rate suggests that a single survivor is consistent with luck.
Another defence is forward testing, which provides fresh data untouched by selection. Published, pre-registered tests do the same job.
An honest research log, including failures, is the best protection.
One more place it hides: the pool of strategies visible online. Those still being promoted are the ones that survived long enough to be written about. A thousand failed ones disappeared quietly, and no one compiled the statistics. The visible sample is not a representative one.
What fraction of tested ideas survives forward testing, and what does that say about the backtest winners?
When a researcher builds dozens of strategies, keeps the ones that work, and discards the rest, the surviving set looks impressive. The discarded failures are invisible, so the success rate of the process is overstated. Anyone reading the final report sees only the winners.
The same applies to data. Backtests on instruments that exist today ignore those that were delisted, or in forex, brokers or pairs that no longer trade. Indicators that survived years of use are those that happened to work, and the ones that were dropped aren't in the textbooks.
Defence starts with counting: how many ideas were tested, and how many survived? A one-in-twenty success rate suggests that a single survivor is consistent with luck.
Another defence is forward testing, which provides fresh data untouched by selection. Published, pre-registered tests do the same job.
An honest research log, including failures, is the best protection.
One more place it hides: the pool of strategies visible online. Those still being promoted are the ones that survived long enough to be written about. A thousand failed ones disappeared quietly, and no one compiled the statistics. The visible sample is not a representative one.
What fraction of tested ideas survives forward testing, and what does that say about the backtest winners?