In-Sample and Out-of-Sample: The Only Honest Test
Posted: Sun Oct 04, 2026 10:42 am
In-Sample and Out-of-Sample: The Only Honest Test
When you design a strategy using a set of data, that data is in-sample. Any performance measured on it is inflated, because you shaped the rules to fit it. To know if the strategy works, you need out-of-sample data, which you did not use during design.
The procedure is simple. Split your data, for example, the first 70 percent for development and the last 30 percent held back. Design and refine using only the first part. When finished, run the final rules once on the held-back part.
If performance holds up, with similar win rate and expectancy, that is encouraging. If it collapses, the strategy was likely fitted to noise.
The discipline is in the single use. If you test on the out-of-sample data, dislike the result, adjust, and test again, the data becomes in-sample. The test only works once.
Choosing the split matters. Out-of-sample data should include different market conditions from the development data, so a strategy built during a trending period gets tested in a ranging one.
Forward testing is the ultimate out-of-sample test: trading the rules in real time on demo or small size, where the future is truly unknown.
Practical step: before developing your next idea, set aside the most recent three months of data and promise not to look at it until your rules are final.
When you design a strategy using a set of data, that data is in-sample. Any performance measured on it is inflated, because you shaped the rules to fit it. To know if the strategy works, you need out-of-sample data, which you did not use during design.
The procedure is simple. Split your data, for example, the first 70 percent for development and the last 30 percent held back. Design and refine using only the first part. When finished, run the final rules once on the held-back part.
If performance holds up, with similar win rate and expectancy, that is encouraging. If it collapses, the strategy was likely fitted to noise.
The discipline is in the single use. If you test on the out-of-sample data, dislike the result, adjust, and test again, the data becomes in-sample. The test only works once.
Choosing the split matters. Out-of-sample data should include different market conditions from the development data, so a strategy built during a trending period gets tested in a ranging one.
Forward testing is the ultimate out-of-sample test: trading the rules in real time on demo or small size, where the future is truly unknown.
Practical step: before developing your next idea, set aside the most recent three months of data and promise not to look at it until your rules are final.