Python + MT5: tick latency logger without buying fancy software

[LondonScalper]

Wanted a simple answer to: "Is my feed/terminal slow *today*, or am I coping?" Didn't want another SaaS dashboard.

Stack (boring on purpose):
- Windows VPS
- MetaTrader5 Python package
- `copy_ticks_from` / tick subscribe depending on need
- Append CSV: local_ts, server_ts (if available), bid, ask, spread
- Small script computes inter-arrival gaps + rolling spread stats per symbol

What it's good for:
- Spotting sick VPS days (clock drift, CPU spikes, terminal freeze)
- Comparing London open tick density vs midday
- Catching "why do my fills feel sticky" before I blame strategy

What it's not:
- A full HFT timestamp bible
- Proof your broker is evil (correlation ≠ courtroom evidence)
- A substitute for proper order-roundtrip logging (I do that separately)

If people want, I can paste a minimal logger skeleton (no strategy, just metrics). Keeping it short in the OP so we don't turn this into a 400-line flex.

Are you logging ticks continuously or only around session open?
Python package vs MQL5 tick logger — which do you trust more for timestamps?
Anyone correlating tick gaps with Windows ETW / CPU steal on VPS and actually finding smoking guns?

[PTScalper]

Wanted a simple answer to: "Is my feed/terminal slow *today*, or am I coping?" Didn't want another SaaS dashboard.

Stack (boring on purpose):
- Windows VPS
- MetaTrader5 Python package
- `copy_ticks_from` / tick subscribe depending on need
- Append CSV: local_ts, server_ts (if available), bid, ask, spread
- Small script computes inter-arrival gaps + rolling spread stats per symbol

What it's good for:
- Spotting sick VPS days (clock drift, CPU spikes, terminal freeze)
- Comparing London open tick density vs midday
- Catching "why do my fills feel sticky" before I blame strategy

What it's not:
- A full HFT timestamp bible
- Proof your broker is evil (correlation ≠ courtroom evidence)
- A substitute for proper order-roundtrip logging (I do that separately)

If people want, I can paste a minimal logger skeleton (no strategy, just metrics). Keeping it short in the OP so we don't turn this into a 400-line flex.

Are you logging ticks continuously or only around session open?
Python package vs MQL5 tick logger — which do you trust more for timestamps?
Anyone correlating tick gaps with Windows ETW / CPU steal on VPS and actually finding smoking guns?

Hi all.

This is a beautifully lean stack. Avoiding SaaS bloat to measure raw terminal truth is exactly the right approach—especially when you need to isolate broker lag from infrastructure bottlenecks.

Here is an architectural take on your three questions, followed by the Pine Script implementation.

1. Continuous vs. Session Open Logging

Continuous is strictly better, provided you have a localized log rotation or retention policy.

If you only log around the London or NY opens, you miss the baseline. To mathematically define a "sick VPS day" or a frozen terminal, you need to know what the standard deviation of inter-arrival gaps looks like during the Asian session doldrums. High tick density during session overlap masks micro-stutters; continuous logging lets you build a clean heat map of typical feed behavior 24/5.

2. Python Package vs. MQL5 Tick Logger

For pure timestamp fidelity, MQL5 wins outright.

The MetaTrader 5 Python integration essentially uses an IPC (Inter-Process Communication) bridge. When you pull ticks via Python, you are at the mercy of:

The MT5 terminal pushing data to the socket.

Network stack loopback latency.

Python's Global Interpreter Lock (GIL) and garbage collector.

Python is perfect for rolling stats and appending CSVs, and it will easily catch a massive 500ms terminal freeze. But if you are trying to measure whether an inter-arrival gap is 5ms or 15ms, the Python bridge introduces too much variable jitter. For sub-millisecond truth, a native MQL5 script utilizing EventChartCustom or writing directly via FileWrite is the only way to eliminate observer effect.

[PTScalper]

3. Tick Gaps, ETW, and VPS CPU Steal

Yes, this is where the smoking guns live.

On shared cloud environments, noisy neighbors cause CPU steal (vCPU overprovisioning). By correlating Windows Event Tracing (ETW) with tick gaps, you will frequently find that the MT5 terminal thread gets parked by the hypervisor for 100ms–250ms right as a massive volatility spike hits the network interface.

This completely explains the "sticky fill" phenomenon. It isn't always the broker holding your order; sometimes the VPS literally hasn't granted the terminal the CPU cycles needed to process the incoming tick and trigger your execution script. Monitoring ready-queue time alongside your tick gaps is one of the best diagnostic moves you can make.

[PTScalper]

TradingView / Pine Script Proxy

Because TradingView does not expose true tick-by-tick arrays (the best you get is 1-second data on Premium tiers), we have to proxy feed health.

This Pine Script tracks inter-candle spread anomalies and tick volume density. It flags moments where the broker's feed might be choking (extreme spread widening outside the rolling standard deviation) while mapping tick density to spot liquidity drop-offs.

Code: Select all

//@version=5
indicator("Feed Health & Spread Monitor", overlay=false)

// A proxy for tick density and feed health.
// TV lacks sub-second tick arrays, so we measure spread anomalies 
// and tick_volume on the lowest available timeframe.

length = input.int(20, "Rolling Window")
mult = input.float(2.0, "Anomaly Multiplier")

// Calculate Current Spread (in ticks/points)
current_spread = (ask - bid) / syminfo.mintick

// Rolling Statistics
avg_spread = ta.sma(current_spread, length)
dev_spread = ta.stdev(current_spread, length)

// Anomaly Detection (Spread widens beyond standard deviation)
upper_band = avg_spread + (dev_spread * mult)
is_anomaly = current_spread > upper_band

// Plotting
plot(current_spread, title="Current Spread", color=color.new(color.blue, 0), style=plot.style_line)
plot(avg_spread, title="Avg Spread", color=color.new(color.gray, 0))
plot(is_anomaly ? current_spread : na, title="Spread Spike", color=color.red, style=plot.style_circles, linewidth=2)

// Tick Volume Density (plotted as a histogram underneath)
// Helps spot session open density vs. quiet period drop-offs
plot(volume, title="Tick Volume", color=color.new(color.orange, 70), style=plot.style_columns)

[PTScalper]

How can I monitor and log Windows CPU steal and thread parking from Python to correlate with my MT5 tick gaps?

Method 1: The "Jitter Probe" (Universal & Bulletproof)

The most reliable way to catch a noisy neighbor or hypervisor throttling is to measure thread starvation.

The concept is simple: ask Windows to put a Python thread to sleep for exactly 1 millisecond. When the thread wakes up, calculate the elapsed time. If it took 45 milliseconds instead of 1ms, your VPS was just parked by the hypervisor, and your MT5 terminal was essentially frozen for that exact duration.

Note: Windows defaults to a 15.6ms system timer resolution. We have to use ctypes to force the Windows kernel to 1ms resolution, or this won't work.

Code: Select all

import time
import ctypes
import threading
import csv

# Force Windows system timer to 1ms resolution
winmm = ctypes.windll.winmm
winmm.timeBeginPeriod(1)

def run_jitter_probe(csv_path="cpu_steal_log.csv", threshold_ms=10.0):
    """
    Sleeps for 1ms. If it takes longer than threshold_ms to wake up, 
    the hypervisor stole our CPU cycles (or the OS thread queue is choked).
    """
    with open(csv_path, 'a', newline='') as f:
        writer = csv.writer(f)
        writer.writerow(["local_ts", "requested_sleep_ms", "actual_sleep_ms", "stolen_ms"])
        
        while True:
            start = time.perf_counter()
            time.sleep(0.001)  # Yield CPU for 1ms
            end = time.perf_counter()
            
            actual_ms = (end - start) * 1000.0
            
            # If actual time exceeds our 1ms request by more than the threshold, log it
            if actual_ms > threshold_ms:
                stolen = actual_ms - 1.0
                writer.writerow([time.time(), 1.0, round(actual_ms, 2), round(stolen, 2)])
                f.flush()

# Run it as a daemon alongside your MT5 tick logger
probe_thread = threading.Thread(target=run_jitter_probe, daemon=True)
probe_thread.start()

[PTScalper]

Why this perfectly correlates with MT5: When a network packet (a tick) hits the VPS network interface, the OS has to wake up the MT5 thread to process it. If the Jitter Probe is experiencing a 50ms delayed wake-up, MT5 is experiencing that exact same 50ms delay between the tick arriving and copy_ticks_from acknowledging it.

Method 2: Native Performance Counters (Hyper-V / KVM)

If you want actual OS-level metrics to log alongside your ticks, you can use the built-in wmi library (install via pip install WMI).If your VPS is running Hyper-V, you can query the holy grail of virtualization metrics: CPU Wait Time Per Dispatch. This measures the exact nanoseconds a virtual CPU spent waiting in the hypervisor queue before being allowed to run on a physical core.

If your VPS is on KVM/Virtio, that counter won't exist. Your best proxy is Processor Queue Length, which tracks how many threads are stuck in the "Ready" state waiting for a core to free up.

Code: Select all

import wmi
import time

def log_windows_counters():
    c = wmi.WMI()
    
    while True:
        # --- FOR KVM/GENERIC VPS ---
        # Look for queue bottlenecks. Consistently > 2 per vCPU is bad.
        sys_perf = c.Win32_PerfFormattedData_PerfOS_System()[0]
        print(f"Processor Queue Length: {sys_perf.ProcessorQueueLength}")

        # --- FOR HYPER-V VPS ---
        # If your VPS is Hyper-V, uncomment below to get exact hypervisor steal.
        '''
        try:
            hv_stats = c.Win32_PerfFormattedData_HvStats_HyperVHypervisorVirtualProcessor()
            for vcpu in hv_stats:
                if vcpu.Name != "_Total":
                    print(f"vCPU {vcpu.Name} Steal Time: {vcpu.CPUWaitTimePerDispatch}")
        except Exception:
            pass # Counter not available on non-Hyper-V hosts
        '''
        
        time.sleep(1) # Poll every second

[PTScalper]

The Correlation Strategy

The ultimate diagnostic move is merging your MT5 CSV and your Jitter Probe CSV by the local_ts timestamp.

Look for a row in your MT5 CSV where the inter-arrival gap is unnaturally large (e.g., a 250ms gap during the London Open when you'd expect a 5ms gap).

Cross-reference that exact second in the Jitter Probe CSV.

If the Jitter Probe fired a stolen_ms warning of ~200ms at that exact moment, you have your smoking gun. The broker didn't freeze the feed; your VPS provider oversold the host, the hypervisor parked your VM, and MT5 physically couldn't process the tick.

[PTScalper]

When executing high-volume scalping strategies, millisecond drift is unacceptable. To isolate the broker's matching engine from your VPS network stack, you need to measure the exact blocking time of a synchronous OrderSend call.

This minimal MQL5 wrapper is designed to drop directly into your existing EA. It uses GetMicrosecondCount() for high-resolution timing and immediately flushes the output to a CSV so your Python script can ingest it in real-time.

Code: Select all

//+------------------------------------------------------------------+
//| Minimal Execution Latency Logger                                 |
//+------------------------------------------------------------------+
#property strict

int csv_handle = INVALID_HANDLE;

// 1. Initialize the file handle once to avoid I/O bottlenecks during live trading
int OnInit() {
    // FILE_COMMON saves to the shared MT5 directory, making it easy for Python to access
    csv_handle = FileOpen("execution_latency.csv", FILE_WRITE|FILE_CSV|FILE_ANSI|FILE_COMMON);
    
    if(csv_handle != INVALID_HANDLE) {
        FileSeek(csv_handle, 0, SEEK_END);
        if(FileSize(csv_handle) == 0) {
            FileWrite(csv_handle, "local_ts,symbol,type,latency_ms,slippage_points");
            FileFlush(csv_handle);
        }
    }
    return(INIT_SUCCEEDED);
}

void OnDeinit(const int reason) {
    if(csv_handle != INVALID_HANDLE) {
        FileClose(csv_handle);
    }
}

// 2. Wrap your execution logic to capture pure roundtrip time
bool ExecuteWithLatencyLog(MqlTradeRequest &request, MqlTradeResult &result) {
    
    ulong start_time = GetMicrosecondCount();
    
    // Synchronous OrderSend blocks the thread until the broker server confirms
    bool success = OrderSend(request, result);
    
    ulong end_time = GetMicrosecondCount();
    
    // Convert microseconds to precise milliseconds
    double latency_ms = (end_time - start_time) / 1000.0;
    
    if(csv_handle != INVALID_HANDLE) {
        datetime local_ts = TimeLocal(); 
        
        // Calculate actual slippage (Requested Price vs Fill Price)
        double slippage = 0;
        if(success && result.deal != 0 && request.price > 0) {
             slippage = MathAbs(request.price - result.price) / SymbolInfoDouble(request.symbol, SYMBOL_POINT);
        }
        
        string action_type = EnumToString(request.type);
        
        FileWrite(csv_handle, 
                  IntegerToString(local_ts), 
                  request.symbol, 
                  action_type, 
                  DoubleToString(latency_ms, 3), 
                  DoubleToString(slippage, 1)
        );
        
        // FileFlush forces the OS to write to disk immediately without closing the handle.
        // This allows your Python daemon to read the CSV concurrently without lock collisions.
        FileFlush(csv_handle); 
    }
    
    return success;
}

[PTScalper]

How to Correlate the Data

By using FILE_COMMON, the CSV drops into Terminal\Common\Files. You can point your Python Pandas/CSV ingestion script to this exact directory.

When you experience a slow fill or high slippage, pull the local_ts from this CSV and cross-reference it with the stolen_ms from the Python Jitter Probe.

If latency_ms is 250ms and stolen_ms is 0, your broker's matching engine or liquidity provider was genuinely lagging.

If latency_ms is 250ms and stolen_ms is 220ms, your broker executed the trade in 30ms, but your Windows VPS hypervisor parked your terminal thread for the rest of the time.