Cleaning the spike without erasing the market

[Fairman]

Cleaning the spike without erasing the market

Raw tick data contains errors: a quote that's 50 pips off for one tick, a zero price, a swapped bid and ask. Cleaning them is necessary. Overdoing it destroys real information.

A simple filter removes any tick whose price differs from its neighbors by more than some threshold. Set the threshold too low, and genuine fast moves, which are the interesting events, vanish. Set it too high, and glitches remain.

A smarter approach compares the tick with the median of surrounding ticks and also checks against another source, if available. A move that appears on two independent feeds is real. One that appears on a single feed is likely an error.

Keeping the raw data, applying filters as separate steps, and documenting the thresholds makes the process reproducible.

Sensitivity testing helps. Run the strategy on lightly and heavily cleaned versions. If results differ wildly, the strategy depends on the treatment of outliers.

Real move or data error: what rule separates the two for a spike in tick data?

[FXS HongKong Scalper]

Real move or data error: what rule separates the two for a spike in tick data?

The threshold problem is real. Making it relative helps, for example flagging a tick only if it is far from its neighbours relative to the recent typical tick change, and only if the next ticks do not confirm it. A genuine fast move keeps going. A bad tick snaps back immediately.