Out-of-sample testing at very short timeframes is harder than it looks

[Shadow Trader]

The textbook method: optimize on one period, test on another. If results hold, the strategy is real. For scalping, the second step often fails for reasons beyond overfitting.

Market microstructure changes. Spreads in 2019 weren't the same as in 2023. Liquidity providers' behavior evolves, brokers update execution technology, and the character of tick data shifts. A strategy tuned on old data might fail not because it was overfitted but because the market's structure moved.

There's also the issue of data volume. Short-term strategies produce many trades, so a few months of data may be enough for statistical significance. But few months of data may represent only one regime. The sample is large in trades and tiny in market conditions.

Walk-forward testing helps: optimize on a window, test on the next, slide forward, repeat. It shows how stable the parameters are over time.

And the real out-of-sample test remains live trading, with minimum size, for long enough to experience varied conditions.

How long should a forward test run on a scalping strategy before the results can be trusted?