Survivorship in strategy ideas: the ones that never got posted

[Shadow Trader]

Forums are full of strategies that work. Screenshots of equity curves. Testimonials. A neat and consistent picture. What's missing is the graveyard.

For every strategy that gets posted, a hundred others were tried and dropped. The ones shown are the survivors, and in a large enough pool of random strategies, some will look brilliant by chance alone. That's selection bias.

Think of coin flippers. Start with a thousand people each flipping ten times. A few will get ten heads. Those people will look like geniuses and attract followers. Their success says nothing about skill.

The same applies to optimized backtests. Test enough variations and a few will fit the past perfectly. Presented alone, they look like discoveries.

A sobering exercise is to count how many ideas were tested before the current one. If the answer is fifty, the effective significance of the current result needs to be discounted heavily.

Strategies that work for a reason, such as a structural flow or a known microstructure effect, are better candidates. A reason doesn't guarantee success, but absence of any reason is a warning.

How many variations were tried before settling on the strategy in use?