How to properly backtest your own forex scalping strategy?

[FTtrader]

Hi all,

the most sophisticated way to simulate high-frequency trading (HFT) and scalping strategies in Forex is using a Discrete Event Simulation (DES) with a Limit Order Book (LOB) reconstruction.

Standard backtesting tools (like MetaTrader or simple Python libraries using OHLCV bars) fail for scalping because they cannot accurately model the three biggest realities of micro-trading: latency, slippage, and queue position.

Here is a breakdown of what makes an advanced simulation, followed by a Python framework to get you started.

The Anatomy of an Advanced Simulation
To realistically simulate forex scalping, your engine must model:

Market-By-Order (MBO) Data: You need Level 3 (or at least tick-by-tick Level 2) order book data, not just bid/ask spreads.

Order Queue Position: If you place a limit order at the bid, you don't get filled just because a market sell order hits that price. You get filled when all the orders ahead of you in the queue are filled or canceled.

Network Latency: The delay between your algorithm seeing an event and your order reaching the exchange.

Market Impact: When your algorithm hits the market with size, it consumes liquidity, widening the spread and moving the price against you (slippage).

[FTtrader]

The HFTBacktest Framework
Building an order book engine from scratch that handles queue position and latency correctly is exceptionally complex and computationally heavy.

Instead of starting from zero, the standard approach in Python is to use hftbacktest. It is currently one of the most advanced open-source frameworks specifically designed for this. It is written in Rust for speed but has a Numba-compiled Python API.

Here is how to set up a basic scalping simulation that accounts for queue position and latency.

1. Installation
First, install the library:

Code: Select all

pip install hftbacktest

2. A Basic Scalping Simulator
This script demonstrates how to set up the environment, define a simple liquidity-taking rule, and simulate it against tick data.

Code: Select all

import numpy as np
from hftbacktest import BacktestAsset, HashMapMarketDepthBacktest, BUY, SELL, GTC, GTX
from hftbacktest.stat import Stat
from numba import njit

# 1. Define the scalping logic using Numba JIT for speed
@njit
def scalping_algo(hbt):
    asset_no = 0
    tick_size = hbt.depth(asset_no).tick_size
    lot_size = hbt.depth(asset_no).lot_size
    
    # Track state
    position = 0.0
    
    while hbt.elapse(10_000_000): # Elapse 10ms at a time
        
        # Clear out completed/canceled orders
        hbt.clear_inactive_orders(asset_no)
        
        # Get the current state of the order book
        depth = hbt.depth(asset_no)
        best_bid = depth.best_bid
        best_ask = depth.best_ask
        
        if np.isnan(best_bid) or np.isnan(best_ask):
            continue
            
        # Calculate book skew (imbalance)
        bid_vol = depth.bid_depth
        ask_vol = depth.ask_depth
        
        # Simple Logic: If the bid is heavily stacked (buying pressure), go long
        # Note: This is an example. Real scalping logic requires much deeper feature engineering
        if bid_vol > ask_vol * 3 and position == 0:
            # We are crossing the spread (taking liquidity)
            # Send an aggressive order to the ask price
            hbt.submit_buy_order(
                asset_no, 
                order_id=1, 
                price=best_ask, 
                qty=lot_size, 
                time_in_force=GTC
            )
            position += lot_size
            
        # If we have a position and the book flips, exit
        elif ask_vol > bid_vol * 1.5 and position > 0:
            hbt.submit_sell_order(
                asset_no, 
                order_id=2, 
                price=best_bid, 
                qty=lot_size, 
                time_in_force=GTC
            )
            position = 0.0

# 2. Configure the simulation environment
def run_simulation(data_file):
    # Setup the asset with specific exchange rules
    asset = (
        BacktestAsset()
        .data([data_file]) # Requires formatted tick data
        .initial_position(0.0)
        .linear_asset(1.0)
        # Model latency: assume 5ms constant feed latency, 10ms order latency
        .constant_latency(feed_latency=5_000_000, order_latency=10_000_000) 
        # Queue position model: Uses standard probabilistic queue modeling
        .prob_queue_model() 
    )
    
    # Initialize the backtester
    hbt = HashMapMarketDepthBacktest([asset])
    
    # Run the compiled algorithm
    scalping_algo(hbt)
    
    return hbt

# NOTE: To run this, you need L2/L3 tick data formatted as np.ndarray
# e.g., run_simulation('eurusd_2024_tick_data.npz')

Why this approach is necessary:The constant_latency model: Ensures your algorithm doesn't "see" a price update until 5ms after it happened, and your order doesn't arrive at the exchange until 10ms later. In reality, the price often moves away during that 15ms window.
The prob_queue_model: When you place limit orders, this mathematically estimates where you are in the line based on trade flow, preventing the classic backtesting error of assuming you got filled at the exact high/low of a candle.

[FTtrader]

If you like it or dislike it please let me know, thanks.

Have a nice day.

[LondonScalper]

Standard backtesting tools (like MetaTrader or simple Python libraries using OHLCV bars) fail for scalping because they cannot accurately model the three biggest realities of micro-trading: latency, slippage, and queue position.

Correct diagnosis. Bar backtests flatter M1 systems; the question is how much complexity you need for your decision.

Full LOB/DES reconstruction is ideal for maker/HFT research. For discretionary or simple stop-entry scalping on a retail path, I get more truth from: tick data from the same broker path, conservative slip assumptions, spread regimes by session, and a journal of live rejects. Perfect foreign books can lie politely.

If you go the hftbacktest route, keep the research book and the execution venue tagged separately — otherwise you optimise for a market you do not trade.

Are you validating a limit-queue strategy, or mainly trying to stop OHLC fantasy fills on stop entries?

[LondonNewsTrader]

Standard backtesting tools (like MetaTrader or simple Python libraries using OHLCV bars) fail for scalping because they cannot accurately model the three biggest realities of micro-trading: latency, slippage, and queue position.

This is the uncomfortable truth behind most "90% win rate" M1 exports.

OHLCV will cheerfully fill you inside a bar that never offered your price once spreads widened into the print. If your edge is a few pips on majors, you need at least tick realism and a pessimistic slip assumption around NFP/CPI. Full LOB reconstruction is ideal; a disciplined stress sheet with latency and p95 slip is the practical middle for most desks.

I still run bar tests for structure research, then discard anything that dies when I add 2–4 ticks of adverse friction and a news blackout. Frameworks like hftbacktest are the right direction when the strategy claims true queue-sensitive entries.

What minimum data grade do you accept before calling a scalp backtest decision-grade — tick, or L2/L3 only?