The out-of-sample set that gets used up
Posted: Sat Oct 31, 2026 2:11 pm
The out-of-sample set that gets used up
Reserving a portion of data for out-of-sample testing is standard. The rule is to look at it once. Each time it's used to decide between versions, a little of its independence is spent.
A trader tries a strategy, tests it on the hold-out period, tweaks it, tests again, and repeats. After twenty rounds, the hold-out data has shaped the strategy as much as the training data.
The result is a model that looks validated but has effectively been fit to everything.
The discipline is hard, because the temptation to improve is strong. A related approach is to collect fresh data going forward: forward testing on a demo or tiny live account provides a truly unseen sample.
Time is the cleanest out-of-sample set. The market produces new data every day, and it can't be tuned on in advance.
One hold-out period reused to pick a strategy version: how many times so far?
Reserving a portion of data for out-of-sample testing is standard. The rule is to look at it once. Each time it's used to decide between versions, a little of its independence is spent.
A trader tries a strategy, tests it on the hold-out period, tweaks it, tests again, and repeats. After twenty rounds, the hold-out data has shaped the strategy as much as the training data.
The result is a model that looks validated but has effectively been fit to everything.
The discipline is hard, because the temptation to improve is strong. A related approach is to collect fresh data going forward: forward testing on a demo or tiny live account provides a truly unseen sample.
Time is the cleanest out-of-sample set. The market produces new data every day, and it can't be tuned on in advance.
One hold-out period reused to pick a strategy version: how many times so far?