Roundup: Overfitting and optimisation, plateaus, out-of-sample and walk-forward in one place
Posted: Thu Oct 15, 2026 4:44 pm
Roundup: Overfitting and optimisation, plateaus, out-of-sample and walk-forward in one place
Overfitting has seven threads on the algo board, from definitions to a composite story about a trader who tuned everything. They agree on almost all of it, and the replies add some very practical checks, so here's the combined version.
How it happens
The overfitting thread describes the slow version: test an idea, see losses, add a filter, add another rule for a specific bad period, adjust settings until the curve looks good. Each step feels reasonable. Together they produce a strategy built to explain the past. The curve fitting thread shows the fast version: the usual indicator setting loses, change the period from 14 to 11 and the curve looks beautiful. That's not a discovery, it's fitting noise.
The over-optimised story thread has a reply that spells out the arithmetic. Give five parameters ten values each and you're testing 100,000 versions. The top one will look great even on random data.
A sneaky one: time filters
A reply in the curve fitting thread names a parameter people overfit without noticing: the time window. "Only trade between 08:15 and 10:40" looks like a session rule, but if those times were found by optimisation, it's a fitted parameter like any other.
Plateaus, not peaks
The optimisation thread and the plateaus thread make the same core point. If a 20-period average wins, look at its neighbours, say 15 to 25. When those hold up too, the idea holds and the exact number barely matters. If only 20 works, it was luck. Pick a value in the middle of the stable region, not at the top of the spike.
Quick checks from the replies:
The in-sample/out-of-sample thread suggests splitting the data, say 70 percent for development and 30 held back, and testing on the held-back part only once. A reply admits the hardest part is that "only once" rule. Peeking at the held-back data just to check turns it into in-sample data. One trick: keep it in a separate file you don't open until the end.
Walk-forward
When data is limited, walk-forward is the middle ground. The walk-forward windows thread gives a starting structure: in-sample long enough for 200 or more trades but short enough to reflect current conditions, out-of-sample around 20 to 30 percent of that, rolled forward. Its example is 6 months in, 2 months out, rolled monthly over three years. More steps give a more reliable picture.
The common rule
Fewer parameters, wider plateaus, and a test period you haven't touched. Every thread here comes back to those three.
How many parameters does your main EA have, and have you ever cut some and found it held up better?
Threads this pulls from
Overfitting has seven threads on the algo board, from definitions to a composite story about a trader who tuned everything. They agree on almost all of it, and the replies add some very practical checks, so here's the combined version.
How it happens
The overfitting thread describes the slow version: test an idea, see losses, add a filter, add another rule for a specific bad period, adjust settings until the curve looks good. Each step feels reasonable. Together they produce a strategy built to explain the past. The curve fitting thread shows the fast version: the usual indicator setting loses, change the period from 14 to 11 and the curve looks beautiful. That's not a discovery, it's fitting noise.
The over-optimised story thread has a reply that spells out the arithmetic. Give five parameters ten values each and you're testing 100,000 versions. The top one will look great even on random data.
A sneaky one: time filters
A reply in the curve fitting thread names a parameter people overfit without noticing: the time window. "Only trade between 08:15 and 10:40" looks like a session rule, but if those times were found by optimisation, it's a fitted parameter like any other.
Plateaus, not peaks
The optimisation thread and the plateaus thread make the same core point. If a 20-period average wins, look at its neighbours, say 15 to 25. When those hold up too, the idea holds and the exact number barely matters. If only 20 works, it was luck. Pick a value in the middle of the stable region, not at the top of the spike.
Quick checks from the replies:
- Shift each setting by 10 to 20 percent in both directions and test again. Genuine edges degrade gently, overfit ones collapse.
- If moving one parameter from 14 to 13 or 15 collapses the results, don't trust it.
- Use a two-parameter heatmap where possible. What you're hoping to see is a wide patch of decent results, not a single bright cell surrounded by bad ones.
- Set a minimum trade count before optimising, otherwise the "best" result is often a setting that took 15 trades and got lucky.
- Any new rule has to make sense on its own before you look at whether it improves the test.
The in-sample/out-of-sample thread suggests splitting the data, say 70 percent for development and 30 held back, and testing on the held-back part only once. A reply admits the hardest part is that "only once" rule. Peeking at the held-back data just to check turns it into in-sample data. One trick: keep it in a separate file you don't open until the end.
Walk-forward
When data is limited, walk-forward is the middle ground. The walk-forward windows thread gives a starting structure: in-sample long enough for 200 or more trades but short enough to reflect current conditions, out-of-sample around 20 to 30 percent of that, rolled forward. Its example is 6 months in, 2 months out, rolled monthly over three years. More steps give a more reliable picture.
The common rule
Fewer parameters, wider plateaus, and a test period you haven't touched. Every thread here comes back to those three.
How many parameters does your main EA have, and have you ever cut some and found it held up better?
Threads this pulls from
- Curve Fitting: How Indicator Settings Become a Trap
- Overfitting: When a Strategy Is Built to Explain the Past
- Optimizing Parameters Without Fooling Yourself
- Parameter Plateaus: Why a Stable Region Beats the Best Single Setting
- In-Sample and Out-of-Sample: The Only Honest Test
- Walk-Forward Windows: Choosing In-Sample and Out-of-Sample Lengths
- The Trader Who Over-Optimized Everything (Story)