forex-scalping.com

Spot the Overfit in a Parameter Table

A trader tests a moving average period on a strategy and reports net results in R over 200 trades for each setting. Period 8: plus 4. Period 9: minus 6. Period 10: plus 2. Period 11: plus 48. Period 12: minus 5. Period 13: plus 1. Period 14: minus 3.

Which one looks best? Period 11, with plus 48R. The trader wants to adopt it.

Look at the neighbors. Period 10 gives plus 2, and period 12 gives minus 5. A change of one unit in either direction collapses the result from plus 48 to roughly breakeven or a loss. The overall pattern jumps around between negative and slightly positive values, with one spike.

That is the signature of noise. A robust idea shows a plateau: periods 9 through 13 would all be in a similar range, and the exact choice would matter little. A single peak surrounded by mediocre values is likely the result of fitting.

What to do? Do not use period 11. Consider whether the strategy has an edge at all, since the median result across settings is close to zero. If you want to continue, test on out-of-sample data with a fixed setting chosen before seeing results.

Compute the average: 4 minus 6 plus 2 plus 48 minus 5 plus 1 minus 3 is 41, divided by 7, about 5.9R, but without the spike it is minus 1.2R on the other six.

Takeaway for your own testing: always print the full table, never just the best row.

Exercise: look at your last optimization and check how many neighbors of your chosen setting are profitable.

A textbook overfit. A robust parameter should have neighbours that also perform reasonably well. If the results jump around like this, the strategy probably has no real edge for that parameter at all, and the plus 48 is noise.

The table is a textbook example of overfitting. Period 11 at plus 48R sits between period 10 at plus 2 and period 12 at minus 5. No plausible market mechanism would make an 11 period average dramatically better than a 10 or 12 period one. The spike almost certainly reflects a few trades that happened to fall favourably with that exact setting. A trader who adopted period 11 would likely see results closer to the neighbours' average in live trading, which is around zero. The full table reveals what the best row alone would hide.

I checked my last optimization after reading this and it was humbling. It was a one minute channel breakout where I tested lookbacks from 10 to 30 in steps of 2, so eleven settings. Eight were profitable, the median was about plus 9R, and the best row was a lookback of 22 at plus 44R. Its neighbors were 20 at plus 3R and 24 at minus 2R, the same lonely spike as in this table.

I chose 16 instead, at plus 17R, with neighbors at plus 12R and plus 14R. Nothing special about the number, it just sat in the middle of a plateau. Running it live for three months on a small account gave a result around the median of the table, not near the best row, which is what I expected.

My rule now is to average each setting with its two neighbors before ranking them, so one lucky row cannot win.

The average without the spike is the most telling number. Across the other six settings, the average is about minus 1.2R over 200 trades, essentially zero. That suggests the strategy has no edge at any setting, and the plus 48R result is a statistical accident. With seven settings tested, finding one that looks good by chance is quite likely. The more settings tested, the more likely a spike appears. This connects to the counting tests thread, where testing many combinations makes the best result increasingly unreliable as evidence of a real edge.

A useful test is to check whether the period 11 result is driven by a small number of trades. Sorting the 200 trades by R and looking at the top few might show that a handful of large winners account for most of the 48R. If those trades would have been taken with nearby settings too but exited differently, the spike might come from exit timing rather than entry. Understanding why the result is so different from its neighbours often confirms that it's noise. If no sensible reason can be found, that's itself strong evidence against using the setting.

Printing the full table, as the takeaway says, is a simple habit that prevents a lot of self deception. Plotting the results as a chart, with the parameter on one axis and the result on the other, makes plateaus and spikes obvious at a glance. A strategy worth trading usually shows a broad region of positive results, where small parameter changes make little difference. Has anyone here found a spike like this in their own testing and resisted the temptation to use it? I'd be interested in what the strategy did on out of sample data afterwards.