Out-of-sample testing on tick data: where the common methods break down

[Shadow Trader]

Splitting data into a training period and a test period is standard practice. For scalping on tick data, a few quirks make it trickier.

Market behaviour at the microstructure level changes over time. Tick sizes get revised, liquidity providers come and go, and brokers alter their feeds. A strategy tuned on last year's ticks may be fitting to a feed that no longer exists.

Autocorrelation within the data means that adjacent periods aren't independent. Testing on the week right after the training window can be too optimistic, because conditions are very similar.

A more honest approach is walk-forward testing: optimise on a rolling window, test on the next slice, move forward, repeat. The results across all test slices give a better picture.

It also helps to test across different market regimes, such as high-volatility months and quiet ones, and to leave a final period completely untouched until the end.

If results collapse in the later periods, the edge was probably in the data and not in the market.

Parameter stability is another useful check. A strategy that works only for one exact setting is probably fitted. One that works across a neighbourhood of settings is more believable.

What has been the most reliable way to separate a real edge from an over-fitted one?