Parameter sweeps: how to avoid fooling yourself

[Shadow Trader]

Running a sweep across hundreds of parameter combinations and picking the best one is how most overfit strategies are born. The best result in a sweep of 500 is, statistically, partly luck.

Some habits make sweeps more honest. Look at the whole landscape, not the peak: a good region is a broad plateau where neighbouring values also work. A lone spike surrounded by losers is almost certainly noise. Keep the number of parameters small, since each extra one multiplies the room for fitting. Use out-of-sample data that was never touched during the sweep.

Another trick is the "random baseline": run the same sweep on shuffled or randomized price data. The best result from random data shows how good a purely lucky result can look. If the real strategy's peak doesn't clearly beat that, there's no edge to speak of.

Choosing the middle of a plateau, not the top of a spike, usually produces more durable settings.

Does the process include an out-of-sample test or a randomized baseline, and how often does the best parameter set survive it?