Overfitting at tick level: more data, more ways to fool yourself

[Fairman]

Overfitting at tick level: more data, more ways to fool yourself

Tick data feels like precision. Millions of data points, every quote in order. It also offers far more room for a strategy to find patterns that are not there.

With enough parameters, any dataset can be mined for a setup that looks wonderful. A rule about the third tick after a 0.4 pip move, combined with a spread condition and a time filter, can fit the past perfectly and mean nothing.

Warning signs include extremely specific rules, many parameters, results that collapse when parameters are nudged slightly, and a very large number of trades with a tiny average profit that disappears once costs rise by a fraction.

Remedies are standard but often skipped. Hold out a portion of the data and test on it once. Test on a different period, a different pair, and a different broker's data. Check that results change gradually rather than abruptly when parameters change.

A strategy whose edge depends on exact values and exact data is probably a story about the past, not a rule about the future.

A good habit is counting the parameters. Every extra rule, threshold, and filter is a chance to fit noise. Strategies with three or four parameters that behave sensibly across a range of values tend to survive. Those with a dozen tuned to the decimal rarely do.

Another is demanding a reason. A pattern with a plausible mechanism, such as a flow that happens at a known time, is more believable than one that simply appeared in the data. The mechanism does not guarantee it will persist, but its absence is a warning.

A related habit is keeping a list of ideas tried and discarded. Counting how many variations were tested before one looked good puts the result in context, since the best of fifty attempts is not the same thing as a single clean hypothesis.

What out-of-sample tests have been used, and have they ever killed a favourite idea?