forex-scalping.com

Missing bars, holidays and gaps in Python backtests

When you load FX data into Python, you rarely get a perfectly regular series of bars. Weekends are missing, holidays have short or no sessions, and data sources have gaps where a feed dropped. If your backtest assumes every bar exists, those gaps cause subtle bugs.

The most common one is with indicators that use a fixed number of bars. A 20 bar moving average across a weekend gap covers Friday evening and Sunday night as if they were consecutive. That's usually fine. But if your logic uses time, like 'exit after 60 minutes', and you count bars instead of minutes, a gap can make the exit happen much later in real time than intended.

Resampling is another trap. Resampling M1 data to M5 with pandas can create empty bars for periods with no data, filled with NaN. If you forward fill them, you create fake flat candles. If you drop them, your bar count changes. Either way, it's important to choose deliberately.

Session logic also suffers. If your strategy trades the London open and a holiday has no data in that window, the code should skip the day rather than trade the first bar it finds.

I'd check the data for gaps before running anything, list them, and decide how each type should be handled.

A simple gap check is to compute the time difference between consecutive bars and list any gap larger than the bar size, outside of the normal weekend. That shows feed outages and holidays. You can compare them with a holiday calendar to tell which is which. Feed gaps during normal trading hours are the ones to worry about, since they might hide real price moves. If there are a lot of them in a data source, it might be worth finding better data before trusting any results. Writing the list of gaps to a file and keeping it with the data also helps when you rerun tests months later and need to remember what was excluded.

Time zones make gaps harder to read. If your data is in broker server time and you convert it to UTC, the weekend gap might shift and appear to start or end at odd times. Holidays like Christmas might show as partial days. Converting everything to one zone first, then checking for gaps, keeps things clear. The pandas tz_convert function handles daylight saving properly as long as the timestamps are timezone aware, which is the main thing to get right before any analysis. Storing everything in UTC internally and only converting for display is the habit that saves the most trouble here.

Forward-filling missing bars is the most common fix and also the most dangerous one, because it creates flat bars that never traded. A strategy that enters on "no movement" conditions can end up trading those fake bars in the test.

A safer approach is to keep the gaps and make the strategy logic aware of them: any indicator window that spans a gap longer than, say, five minutes is marked invalid until it fills with real data again. pandas makes this easy by checking the time difference between consecutive rows.

Platforms download price history from the broker. Sometimes it arrives with holes: missing minutes, missing hours, whole sessions with no data. Charts fill the gaps by connecting the last known price to the next.

Indicators then calculate on bad input. A moving average over a section with missing bars shifts. A volatility measure like ATR spikes or collapses where the gap sits. A signal that looks reliable on a chart might have been produced by a hole.

Backtests have the same flaw, and it's worse because it's invisible. A test period with 3 percent missing bars can produce results that look better or worse than reality.

Checking data quality before testing means comparing bar counts with what a full session should contain. A one-minute chart should have 1,440 bars per day on a 24-hour market, less over weekends. Gaps show up as shortfalls.

Some platforms offer a history center to re-download or import clean data, and third-party tick data sources can fill in the gaps.

How is data quality checked here before trusting a backtest?

Daily candles in FX data can differ depending on the source's day boundary. Some data starts the day at midnight UTC, others at 17:00 New York, and some brokers at midnight in their own server time. If your strategy uses daily highs and lows, results can change depending on which boundary the data uses. Checking the first and last bar of a few days, and comparing with your broker's charts, shows whether the data matches what you'll trade live. Converting to a consistent day boundary in pandas before analysis avoids subtle errors.

How do you handle the first bar after a gap in backtests? If your strategy enters on the open of a bar, and the first bar after the weekend opens with a gap, the fill in the backtest is at the gap price, which might be realistic or not depending on the broker. Some brokers' first prices after the weekend are quite wide. Adding extra spread or slippage on the first bar after a gap, or simply not allowing entries on it, gives a more cautious result. It's a small change but avoids trusting fills that wouldn't happen live.