forex-scalping.com

Pandas resample and the off by one bar when building M5 from M1

If you build higher timeframe bars from M1 data in Python, there's a quiet mistake that can wreck a backtest without throwing any errors. It's about which bar each resampled candle gets labelled with, and which minutes go into it.

In pandas, df.resample('5min') groups by time bins. By default for minute frequencies, the bin includes its left edge and the label is the left edge too. So the 10:00 bar contains the minutes 10:00 to 10:04. That matches how MT4 and MT5 label bars, by their open time.

The trouble starts when your M1 data is labelled by close time, which some vendors do. Then a row stamped 10:01 is really the bar from 10:00 to 10:01, and the default resample shifts everything by one minute. Your M5 bars end up mixing the wrong minutes.

The second trap is joining the M5 back onto the M1 rows for signals. If an M1 row at 10:02 is matched with the 10:00 M5 bar, that bar isn't finished yet at 10:02. Using its close is lookahead. The safe way is to shift the M5 series by one bar before joining, so each M1 row only sees completed M5 bars.

Use first, max, min, last and sum for open, high, low, close and volume, and drop the empty bins from weekends.

Time zones make this worse. If your M1 data is in UTC and you resample to H4 or D1, the bars won't line up with the broker's daily candles, which usually start at 17:00 New York time. Pandas resample has an offset and an origin parameter for this. For daily bars that match most FX brokers, convert the index to New York time and use an offset of 17 hours, or convert to the broker's server time first. Otherwise your daily high and low won't match what you see on the chart.

Missing minutes need care. FX data has gaps on thin hours and around the daily rollover. When a five minute bin has only three M1 bars in it, resample still makes a bar, just from the data that's there. That's usually fine. But if a whole bin is empty, you get a row of NaN that should be dropped. Forward filling the close into empty bins can create fake flat bars that will mess with indicators like ATR. Dropping them is more honest. Calculating ATR on the cleaned bars and on the forward filled ones shows the difference quickly.

A quick sanity check after resampling saves a lot of grief. Pick a few random M5 bars and compare them with the same bars on your broker's chart. Open, high, low and close should match within a point or so, allowing for feed differences. If the times are offset by one bar or the highs are consistently different, something is wrong with the labelling or time zone, and every backtest result built on top of it is suspect until that's fixed. Ten random bars spread over different days and sessions is usually enough to catch it.

For signals that use several timeframes at once, a common clean approach is to compute each higher timeframe separately, shift it by one completed bar, and then reindex it onto the M1 index with forward fill. That way each M1 row sees the last completed value of each higher timeframe and nothing from the future. merge_asof in pandas also works well for this if you set direction to backward. Either approach is fine as long as you test it on a few rows by hand to confirm no future data slips in.

A useful check for the off by one issue is to compare the resampled bars with a broker's own M5 history for a few hours. If every bar is shifted by one interval, the label and closed side settings are the cause. On the gaps, thin hours on FX are mostly the Asian late afternoon and the rollover, so an ATR built from bars with empty bins dropped will still run a little high or low at those times, depending on the pair. Counting the M1 bars inside each M5 bin and keeping that as a column lets you filter thin bars later instead of guessing.

On a stock chart, each bar opens at a different price than the previous close, because trading stops overnight. On a forex chart, the open of each bar equals the close of the previous one, almost always. It looks like a continuous line.

That continuity is partly an illusion. Each bar's open is defined as the first tick of the period. The previous close is the last tick of the earlier period. In a liquid market they're nearly identical, but during fast moves or quiet periods, the first tick of a new bar can differ by a pip or more from the last tick of the previous one.

On many platforms, the open is simply set to the previous close for a clean look, so the gap is hidden. The tick data tells the truth.

It's a minor thing, but a strategy that relies on a precise entry at the bar's open might be assuming a price that the market never offered.

Did the gap between a bar's open and the previous close ever explain a mismatch between backtest and live?