The Trader Who Over-Optimized Everything (Story)

[Fairman]

The Trader Who Over-Optimized Everything (Story)

A composite story based on patterns many traders experience. Names are illustrative.

Let's call him Dayo. He built a strategy, backtested it, and saw decent results. But he wasn't satisfied. He tweaked the entry timing. Then the stop distance. Then the session filter. Then the moving average period.

After every tweak, the backtest looked better. Eventually, it showed an almost perfect equity curve. He felt like he had found the holy grail.

He went live with confidence. In the first month, the strategy lost almost every trade.

Confused, he went back to his data. A friend with a statistics background looked at it and explained gently: "You didn't find an edge. You fitted the past."

Dayo had tuned his rules so tightly to historical noise that they described what had already happened, not what was likely to happen. The strategy was a costume tailored to one specific past, and it fit nothing else.

He returned to a simple version of the original idea, with fewer parameters and fewer filters. The backtest looked worse, but it made sense. Forward testing on demo showed results similar to the backtest.

His takeaway: more rules aren't more intelligence. A strategy that needs perfect conditions to work isn't robust.

If tweaking makes your backtest look too good, be suspicious. Simple, slightly imperfect strategies often survive reality better than polished ones.

Don't fit the past. Prepare for the unknown.

[PropScalpDesk]

The arithmetic behind Dayo's curve is worth spelling out. Five parameters with ten values each is 100,000 combinations. Run them all and the best one will look excellent even on random data, because you've picked the luckiest of 100,000 draws. That's why the live month looked nothing like it.

Two checks catch most of this before money is involved. The first is the plateau test: if a 20-period average works, 18 and 22 should work almost as well. When performance falls off a cliff one step away from the chosen value, you fitted noise. The second is holding back data you never look at while tuning, for example the most recent six months, and running the final version on it exactly once. If you rerun after seeing that result and tweak again, it stops being out-of-sample.

I'd also count trades against parameters. A rule set with eight knobs and 120 trades in the backtest has far too few trades per knob to tell you anything.

[Shadow Trader]

The session filter in Dayo's list is the tweak I trust least, because it hides how many choices were made. Dropping the hour that lost most in the backtest feels like one decision, but choosing it from 24 hours is really 24 tries.

On PropScalpDesk's plateau test, I check neighbours in pairs rather than one parameter at a time. A 20 period average can hold up at 18 and 22 with the stop fixed, then fall apart once the stop moves two pips as well, because the two were tuned together.

So I look at the 3 by 3 grid around the chosen pair of values. If the middle cell is clearly better than the eight around it, I treat it as luck and take the centre of the best region instead.

The holdout has a practical cost for scalpers. Once those six months have been used, they are spent, and the only fresh data left is forward trading, which at three trades a day needs about two months to reach 120 trades.

I have also started keeping a written count of every variant I test. The number is usually embarrassing, and it tells me how much doubt the final backtest deserves.