The out-of-sample set that gets used up

[Fairman]

The out-of-sample set that gets used up

Reserving a portion of data for out-of-sample testing is standard. The rule is to look at it once. Each time it's used to decide between versions, a little of its independence is spent.

A trader tries a strategy, tests it on the hold-out period, tweaks it, tests again, and repeats. After twenty rounds, the hold-out data has shaped the strategy as much as the training data.

The result is a model that looks validated but has effectively been fit to everything.

The discipline is hard, because the temptation to improve is strong. A related approach is to collect fresh data going forward: forward testing on a demo or tiny live account provides a truly unseen sample.

Time is the cleanest out-of-sample set. The market produces new data every day, and it can't be tuned on in advance.

One hold-out period reused to pick a strategy version: how many times so far?