Out-of-sample is not the same as out-of-regime

[Fairman]

Out-of-sample is not the same as out-of-regime

A strategy is optimised on one period and tested on another. It passes. Confidence goes up. Then it fails live.

One reason is that both periods came from the same regime. The market in both had similar volatility, similar liquidity, similar central bank backdrops. The test was out of sample but not out of regime.

For scalpers, regimes include liquidity conditions, typical spread levels, and the dominance of certain participants. A strategy built in a low-volatility environment can break when volatility doubles.

A more demanding test checks performance in different regimes separately. Split the data into calm and volatile periods, high-spread and low-spread periods, trending and ranging periods.

If the strategy only works in one regime, that is fine, as long as it can be identified in real time and the strategy switched off in the others.

Which regimes did the strategy face in its test data, and which have never been seen?