forex-scalping.com

Merging tick data from two sources that disagree on time

If you combine tick or M1 data from more than one source, broker exports, a data vendor, your own logger, the most likely thing to go wrong is time.

Most FX brokers run their servers on a time zone shifted from GMT so that the daily candle starts at 17:00 New York, which often means GMT plus two in winter and plus three in summer. Many data vendors store everything in UTC. Your own logger might use your local machine time. Put them side by side without converting and the bars are off by hours.

Daylight saving makes it worse because the US and Europe change clocks on different dates. For a couple of weeks in March, and again in late October into early November, the gap between New York and London is an hour different from usual. A fixed offset conversion is wrong during those weeks. This year the European change is on the 25th of October and the US one on the 1st of November, so that window is coming up.

The safe approach is to convert everything to UTC using proper time zone rules, not fixed offsets. In Python that means zoneinfo or pytz with a named zone like America/New_York. Do all merging and comparison in UTC. Only convert to broker time at the very end if you need bars that match the platform.

Check the conversion against something you know. A good anchor is a major scheduled release, such as US nonfarm payrolls at 8:30 New York. The big M1 candle should sit at exactly the right UTC minute in every source after conversion. If one source shows the spike an hour off on certain dates, there's a daylight saving issue in that source. Doing this for a handful of releases spread across the year catches nearly every time zone mistake in one go. Rate decisions and CPI releases make good extra anchors because their times are fixed and well known.

Some broker exports don't say what time zone they use at all. The CSV just has times. In that case you have to infer it from the data, usually by looking at when the daily bars start or when the Sunday open happens. A Sunday open at 22:00 UTC in winter and 21:00 UTC in summer suggests a server on New York close time. Write the inferred rule down next to the data file, because six months later you won't remember how you worked it out. A small text file in the same folder with the source name, time zone rule and the date you checked it is enough.

When two sources disagree on prices, not just times, don't average them. Pick one as the primary for each purpose. For backtesting a strategy you trade at a given broker, that broker's data is usually the primary, because it reflects the spreads and quirks you'll actually trade into. Use the vendor data to fill gaps or to sanity check, and flag any rows where the two differ by more than a few points so you can look at them separately. Over time the flagged rows also tell you which source is more reliable during fast markets.

Logging your own ticks in UTC from the start avoids most of this. In MQL5, TimeGMT gives you GMT based on the local machine clock, and TimeTradeServer gives server time. Write both, plus your machine's local time, for the first few days of a new logger. That makes it easy to confirm the offsets and catch a VPS with a wrong clock. A VPS whose clock drifts by a few seconds can make your latency numbers useless without anyone noticing. Most VPS providers let you enable network time sync, and it's one setting to confirm on day one.

Flagging rows where the sources differ is the right approach. A specific case worth checking is the Asian session, especially around the Tokyo open and the rollover. Price differences between sources are often largest when liquidity is thin, and that is also where a backtest's assumptions about fills matter most. Counting the flagged rows per hour of the day shows whether the disagreement is spread evenly or concentrated in thin windows. If it is concentrated, a strategy that trades those hours needs more caution in its results.

There's no single price in spot forex. Every provider's history reflects its own sources, filters, and aggregation. Downloading data from three places gives three slightly different datasets.

Differences are small on average and large at the extremes. Spikes appear in one feed and not another. Spreads differ. Tick counts differ by a factor of two or more. The daily high and low can vary by a pip or two. Around news, discrepancies grow.

This matters for tests. A strategy optimized on one provider's data and then run on a broker whose feed differs can behave differently. The signals shift, entries and exits happen at different times, and stops that were safe become exposed.

The ideal is to test on the same feed that will be traded. Many brokers allow exporting history from their own server. If that's not possible, comparing the strategy's results on two or three external sources shows sensitivity.

If performance varies widely by data source, the strategy is fragile. If it's stable, there's more reason for confidence.

Which data source seems closest to what the broker actually delivers?