Advertisement IC Markets

Cleaning the spike without erasing the market

Share, develop, and backtest custom MQL4/MQL5 Expert Advisors, Python data-scraping scripts, trading bots, and automated market alert systems.
Post Reply
Fairman
Posts: 6955
Joined: Tue Jul 21, 2026 7:11 am
Location: Abuja

Cleaning the spike without erasing the market

Post by Fairman »

Cleaning the spike without erasing the market

Raw tick data contains errors: a quote that's 50 pips off for one tick, a zero price, a swapped bid and ask. Cleaning them is necessary. Overdoing it destroys real information.

A simple filter removes any tick whose price differs from its neighbors by more than some threshold. Set the threshold too low, and genuine fast moves, which are the interesting events, vanish. Set it too high, and glitches remain.

A smarter approach compares the tick with the median of surrounding ticks and also checks against another source, if available. A move that appears on two independent feeds is real. One that appears on a single feed is likely an error.

Keeping the raw data, applying filters as separate steps, and documenting the thresholds makes the process reproducible.

Sensitivity testing helps. Run the strategy on lightly and heavily cleaned versions. If results differ wildly, the strategy depends on the treatment of outliers.

Real move or data error: what rule separates the two for a spike in tick data?
It’s Fairman :geek:
Recommended broker for automated trading & scalping IC Markets
FXS HongKong Scalper
Posts: 543
Joined: Sat Oct 24, 2026 10:15 pm

Re: Cleaning the spike without erasing the market

Post by FXS HongKong Scalper »

Fairman wrote: Real move or data error: what rule separates the two for a spike in tick data?
The threshold problem is real. Making it relative helps, for example flagging a tick only if it is far from its neighbours relative to the recent typical tick change, and only if the next ticks do not confirm it. A genuine fast move keeps going. A bad tick snaps back immediately.
FXS Official Team | Hong Kong Session Desk
Fairman
Posts: 6955
Joined: Tue Jul 21, 2026 7:11 am
Location: Abuja

Re: Cleaning the spike without erasing the market

Post by Fairman »

I would avoid making “continued in the next ticks” the sole test. A real price jump can reverse quickly, while a bad quote can persist for more than one update; that rule risks deleting legitimate volatility in one case and retaining an error in the other. It is safer to flag suspect observations and keep the raw series intact, then apply explicit checks—valid positive prices, bid not above ask where the feed’s conventions make that a data error, monotonic timestamps or documented handling of out-of-order records, duplicates, and values far outside a robust local range. A flagged tick can be compared with nearby quotes and, when available, a genuinely independent feed, but corroboration increases confidence rather than proving which series is correct.

I’d preserve both raw and cleaned versions, log every flag and threshold, and report how each filter changes the specific trades or signals affected. A sensitivity run is useful, but the light/heavy settings should be defined independently of which one gives the best backtest. If a strategy’s performance changes sharply under reasonable cleaning choices, that is a warning that its apparent edge may depend on data quality or on a few extreme observations—not a reason to erase the spike automatically.
It’s Fairman :geek:
Amanhasnoname
Posts: 2060
Joined: Fri Oct 30, 2026 9:45 am

Re: Cleaning the spike without erasing the market

Post by Amanhasnoname »

I learned this lesson by cleaning away the best trades in a backtest. I used a fixed threshold of twenty pips between consecutive ticks to filter glitches on GBPJPY, and the results looked great. Then I noticed that several real news moves had been chopped off, including the ones where my live strategy actually lost money, so the backtest was flattering me by removing exactly the dangerous periods.

What I do now is keep a flag column rather than deleting anything. A tick gets flagged if it sits far outside the median of the surrounding few seconds and the price returns within a couple of updates. Flagged ticks are dropped in one version of the data and kept in another, and I run the strategy on both.

If the two results are close, I stop worrying. If they differ a lot, I go through the flagged ticks by hand and check them against a second feed, which usually only takes half an hour for a month of data.
Post Reply