Out-of-sample testing for people who aren't programmers
Posted: Sat Nov 14, 2026 9:50 am
Out-of-sample testing gets talked about as if it needs code and a big data set. The basic idea is simple enough to do by hand, and it's one of the best protections against fooling yourself with a strategy that only fits the past.
The principle: develop and adjust the rules on one block of data, then test them, unchanged, on a block you haven't looked at.
A manual version:
Pick a period, say three months of M5 charts on one pair, and use only that to develop and refine the rules. Write them down precisely enough that someone else could follow them.
Choose a different period, ideally later in time, that you haven't studied. Use bar replay or simply scroll forward candle by candle without peeking ahead, and record every trade the written rules give, including the ugly ones.
Compare the two periods. Some drop in performance is normal. A collapse usually means the rules were shaped around the specific patterns in the first period.
The hard part is discipline. It's very tempting to tweak the rules after seeing a bad out-of-sample trade, but once you do, that data is no longer out-of-sample. If changes are needed, they need another fresh period to test on.
Forward testing on demo is the same idea moving in real time, and it adds spreads and fills that charts don't show.
Do people here keep a strict separation between the data they build on and the data they test on, or does it tend to blur over time?
The principle: develop and adjust the rules on one block of data, then test them, unchanged, on a block you haven't looked at.
A manual version:
Pick a period, say three months of M5 charts on one pair, and use only that to develop and refine the rules. Write them down precisely enough that someone else could follow them.
Choose a different period, ideally later in time, that you haven't studied. Use bar replay or simply scroll forward candle by candle without peeking ahead, and record every trade the written rules give, including the ugly ones.
Compare the two periods. Some drop in performance is normal. A collapse usually means the rules were shaped around the specific patterns in the first period.
The hard part is discipline. It's very tempting to tweak the rules after seeing a bad out-of-sample trade, but once you do, that data is no longer out-of-sample. If changes are needed, they need another fresh period to test on.
Forward testing on demo is the same idea moving in real time, and it adds spreads and fills that charts don't show.
Do people here keep a strict separation between the data they build on and the data they test on, or does it tend to blur over time?