Cleaning the spike without erasing the market
Posted: Sat Oct 31, 2026 2:02 pm
Cleaning the spike without erasing the market
Raw tick data contains errors: a quote that's 50 pips off for one tick, a zero price, a swapped bid and ask. Cleaning them is necessary. Overdoing it destroys real information.
A simple filter removes any tick whose price differs from its neighbors by more than some threshold. Set the threshold too low, and genuine fast moves, which are the interesting events, vanish. Set it too high, and glitches remain.
A smarter approach compares the tick with the median of surrounding ticks and also checks against another source, if available. A move that appears on two independent feeds is real. One that appears on a single feed is likely an error.
Keeping the raw data, applying filters as separate steps, and documenting the thresholds makes the process reproducible.
Sensitivity testing helps. Run the strategy on lightly and heavily cleaned versions. If results differ wildly, the strategy depends on the treatment of outliers.
Real move or data error: what rule separates the two for a spike in tick data?
Raw tick data contains errors: a quote that's 50 pips off for one tick, a zero price, a swapped bid and ask. Cleaning them is necessary. Overdoing it destroys real information.
A simple filter removes any tick whose price differs from its neighbors by more than some threshold. Set the threshold too low, and genuine fast moves, which are the interesting events, vanish. Set it too high, and glitches remain.
A smarter approach compares the tick with the median of surrounding ticks and also checks against another source, if available. A move that appears on two independent feeds is real. One that appears on a single feed is likely an error.
Keeping the raw data, applying filters as separate steps, and documenting the thresholds makes the process reproducible.
Sensitivity testing helps. Run the strategy on lightly and heavily cleaned versions. If results differ wildly, the strategy depends on the treatment of outliers.
Real move or data error: what rule separates the two for a spike in tick data?