Bad ticks: the single quote that ruins a test

[PTScalper]

3. Zero-Volume and Phased Session Filtering

Rule: Filter or mark quotes where $\text{BidSize} = 0$ or $\text{AskSize} = 0$, or where price quotes update without trade prints during off-hours.

Why: Many data feeds report indicative prices during pre-market, post-market, or rollover hours (e.g., 5:00 PM EST Forex rollover). These quotes have zero executable depth and non-existent liquidity, leading to fictitious fills.

[PTScalper]

4. Mid-Price Jump Validation (Velocity Limiters)

Rule: Reject quotes where the mid-price change $\vert{}\text{Mid}_t - \text{Mid}_{t-1}\vert{}$ exceeds a dynamic return threshold, unless accompanied by a valid traded volume on primary feeds.

$$\text{Mid}_t = \frac{\text{Bid}_t + \text{Ask}_t}{2}$$

[PTScalper]

Python Pipeline Implementation

Here is an institutional-style tick validation class using pandas and numpy:

Code: Select all

import numpy as np
import pandas as pd

class TickDataCleaner:
    def __init__(self, max_spread_mad_multiplier: float = 3.5, max_price_jump_pct: float = 0.02):
        self.max_spread_mad_multiplier = max_spread_mad_multiplier
        self.max_price_jump_pct = max_price_jump_pct

    def clean_quotes(self, df: pd.DataFrame) -> pd.DataFrame:
        """
        Expects df with columns: ['bid', 'ask', 'bid_size', 'ask_size']
        Returns df with valid clean ticks and an 'is_valid' boolean mask.
        """
        clean_df = df.copy()
        
        # 1. Eliminate Crossed & Locked Markets (Bid >= Ask)
        is_crossed_or_locked = clean_df['bid'] >= clean_df['ask']
        
        # 2. Eliminate Zero Liquidity / Depth
        has_zero_depth = (clean_df['bid_size'] <= 0) | (clean_df['ask_size'] <= 0)
        
        # 3. Calculate Spread and Filter Outliers via Rolling MAD
        spread = clean_df['ask'] - clean_df['bid']
        rolling_median_spread = spread.rolling(window=100, min_periods=10).median()
        rolling_mad_spread = (spread - rolling_median_spread).abs().rolling(window=100, min_periods=10).median()
        
        spread_threshold = rolling_median_spread + (self.max_spread_mad_multiplier * 1.4826 * rolling_mad_spread)
        is_abnormal_spread = (spread <= 0) | (spread > spread_threshold)
        
        # 4. Mid-Price Jump Velocity Check
        mid_price = (clean_df['bid'] + clean_df['ask']) / 2.0
        price_jump = (mid_price - mid_price.shift(1)).abs() / mid_price.shift(1)
        is_price_spike = price_jump > self.max_price_jump_pct

        # Combine all invalid flags
        invalid_mask = is_crossed_or_locked | has_zero_depth | is_abnormal_spread | is_price_spike
        
        clean_df['is_valid'] = ~invalid_mask
        
        # Scrub invalid rows (or set to NaN for forward filling)
        clean_df.loc[invalid_mask, ['bid', 'ask']] = np.nan
        clean_df['bid'] = clean_df['bid'].ffill()
        clean_df['ask'] = clean_df['ask'].ffill()
        
        return clean_df

# Example Data Setup
data = {
    'bid': [1.1000, 1.1002, 1.1050, 1.1003, 1.0900, 1.1005],  # 1.1050 crossed ask, 1.0900 spike
    'ask': [1.1002, 1.1004, 1.1040, 1.1005, 1.1002, 1.1007],
    'bid_size': [100, 150, 200, 0, 100, 300],                # 0 size at index 3
    'ask_size': [100, 120, 150, 100, 100, 250]
}
df = pd.DataFrame(data)

cleaner = TickDataCleaner()
cleaned_df = cleaner.clean_quotes(df)
print(cleaned_df)

[PTScalper]

Best Practices for Fill Simulation in BacktestsAvoid Mid-Price Fills:

Never fill backtest orders at the mid-price ($\frac{\text{Bid} + \text{Ask}}{2}$). Always execute buys at the Ask and sells at the Bid.

Apply Slippage Modeling on Spikes: If a real market spike occurs (e.g., during non-farm payrolls), force the backtesting engine to assume maximum slippage rather than giving your strategy the best point on the candle wick.

Pessimistic Matching Engine: If a bad tick creates an extreme wick on a candle, ignore the wick for order fills unless the bar's volume was sufficient to sustain the price move across multiple trades.

[PTScalper]

Translating these data-cleaning and outlier-rejection algorithms into MQL5 is essential when writing robust Expert Advisors (EAs) or custom indicators that ingest historical tick data (MqlTick) or bar history (MqlRates) directly inside MetaTrader 5.

Since MQL5 does not have built-in dataframes like Python’s pandas, the implementation relies on dynamic arrays, custom median sorting functions, and rolling window buffers.

[PTScalper]

MQL5 Implementation: Tick Cleaner & Hampel Filter

Below is a professional MQL5 include-style script (TickCleaner.mqh logic) containing a Hampel filter for price series and real-time tick validation rules for an OnTick() event loop.

Code: Select all

//+------------------------------------------------------------------+
//|                                              TickDataCleaner.mqh |
//|                                  Copyright 2026, Quant Systems   |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------>
#property copyright "Copyright 2026"
#property link      "https://www.mql5.com"
#property version   "1.00"

class CTickDataCleaner
private:
   double            m_max_spread_mad_multiplier;
   double            m_max_price_jump_pct;
   
   // Helper function to calculate median of an array segment
   double ArrayMedian(double &array[], int start, int count)
   {
      if(count <= 0) return 0.0;
      double temp[];
      ArrayResize(temp, count);
      ArrayCopy(temp, array, 0, start, count);
      ArraySort(temp);
      
      int mid = count / 2;
      if(count % 2 == 0)
         return (temp[mid - 1] + temp[mid]) / 2.0;
      else
         return temp[mid];
   }

public:
   CTickDataCleaner(double mad_multiplier=3.5, double price_jump_pct=0.02)
   {
      m_max_spread_mad_multiplier = mad_multiplier;
      m_max_price_jump_pct = price_jump_pct;
   }
   
   ~CTickDataCleaner() {}

   //------------------------------------------------------------------
   // 1. Real-Time Tick Validation (for OnTick)
   //------------------------------------------------------------------
   bool IsTickValid(const MqlTick &current_tick, double last_valid_mid_price)
   {
      // Rule 1: Eliminate Crossed or Locked Markets (Bid >= Ask)
      if(current_tick.bid >= current_tick.ask)
         return false;
         
      // Rule 2: Eliminate Zero Liquidity / Depth
      if(current_tick.volume_real == 0 && current_tick.bid == 0 && current_tick.ask == 0)
         return false;
         
      // Rule 3: Extreme Spread Rejection (Sanity Check)
      double spread = current_tick.ask - current_tick.bid;
      if(spread <= 0 || spread > (current_tick.ask * 0.05)) // Spread > 5% of price is abnormal
         return false;
         
      // Rule 4: Mid-Price Velocity / Jump Check
      if(last_valid_mid_price > 0.0)
      {
         double current_mid = (current_tick.bid + current_tick.ask) / 2.0;
         double jump_pct = MathAbs(current_mid - last_valid_mid_price) / last_valid_mid_price;
         if(jump_pct > m_max_price_jump_pct)
            return false; // Flash spike or bad tick detected
      }
      
      return true;
   }

   //------------------------------------------------------------------
   // 2. Hampel Filter for Historical Array Cleaning
   //------------------------------------------------------------------
   void ApplyHampelFilter(double &price_array[], int window_size, double n_sigmas, double &output_array[])
   {
      int total = ArraySize(price_array);
      ArrayResize(output_array, total);
      ArrayCopy(output_array, price_array);
      
      int half_window = window_size / 2;
      
      for(int i = 0; i < total; i++)
      {
         int start = MathMax(0, i - half_window);
         int end = MathMin(total - 1, i + half_window);
         int count = (end - start) + 1;
         
         // Calculate rolling median
         double med = ArrayMedian(price_array, start, count);
         
         // Calculate MAD (Median Absolute Deviation)
         double abs_diffs[];
         ArrayResize(abs_diffs, count);
         for(int j = 0; j < count; j++)
         {
            abs_diffs[j] = MathAbs(price_array[start + j] - med);
         }
         
         // Median of absolute differences
         double mad = ArrayMedian(abs_diffs, 0, count);
         double threshold = n_sigmas * 1.4826 * mad;
         
         // Check if current point is an outlier
         if(MathAbs(price_array[i] - med) > threshold)
         {
            output_array[i] = med; // Replace outlier with rolling median
         }
      }
   }
};

[PTScalper]

How to Use Inside an Expert Advisor (.mq5)

To integrate this class into your live or backtesting EA, call the validation check inside OnTick():

Code: Select all

CTickDataCleaner *cleaner;
double last_mid = 0.0;

int OnInit()
{
   cleaner = new CTickDataCleaner(3.5, 0.015); // 3.5 MAD, 1.5% max jump
   return(INIT_SUCCEEDED);
}

void OnDeinit(const int reason)
{
   delete cleaner;
}

void OnTick()
{
   MqlTick tick;
   if(!SymbolInfoTick(_Symbol, tick)) return;
   
   // Validate the incoming tick
   if(!cleaner.IsTickValid(tick, last_mid))
   {
      // Drop or ignore the bad tick
      return; 
   }
   
   // Update last valid mid-price and proceed with strategy logic
   last_mid = (tick.bid + tick.ask) / 2.0;
   
   // --- Your Trading Strategy Logic Here ---
}

[Fairman]

Fair point on the Hampel filter, I hadn't thought about using a rolling median instead of a mean, since one fat spike drags the mean toward itself and then hides itself. Where I'd be careful is the window on M1 data. If you run 5 bars centred and a real news candle prints three bars in the same direction, the median can treat the genuine move as an outlier and flatten it. That would make a news-fade system look much better than it is.

Here's how I'd sanity check it. Say your test has 4,000 trades at 0.5 lots and risk of 10 pips each, so about 5 dollars per pip per trade. One phantom 60 pip winner is worth around 300 dollars on its own, which can be more than a whole week of normal results. So I log the filter's flags and look at them by hand before trusting it.

Do you run it on raw ticks or on bars? And did you ever find that the filter removed real moves around NFP?