Where to Get Historical Data and What to Watch For

[Fairman]

Where to Get Historical Data and What to Watch For

Testing requires data, and data quality affects the conclusions. Several sources exist, each with limits.

Brokers and platforms provide historical bar data, often for free, through their trading terminals. Quality varies. Gaps, missing periods, and differences between brokers appear. Data may be adjusted or modeled, especially for older periods.

Charting services like TradingView give access to historical data from various feeds, convenient for visual analysis, though depth of history and tick detail differ.

Third-party data vendors sell tick data and cleaner records, with better coverage and documented methods. Some offer free samples or limited free access, and prices rise with detail.

Exchange data, such as CME currency futures, comes with real volume and is good for studying market microstructure, though it covers futures only.

Central banks and statistical agencies provide free economic data and, in some cases, official exchange rates, useful for macro research.

Things to check include the time zone and the daily close definition, since these change candles. Whether the data includes bid, ask, or mid prices, and whether spread is recorded. The treatment of weekends and holidays. Adjustments for daylight saving.

Compare a sample against a second source to detect errors.

Keep a record of where each data set came from, and its version, so results can be reproduced.

Practical step: compare one week of hourly data for the same pair from two sources, and note any differences in highs and lows.

[Fairman]

The daily close definition is a quiet trap. Data based on New York 5pm and data based on midnight UTC give different daily candles, and therefore different signals. With a midnight UTC close, many brokers also show a small Sunday candle covering the first couple of hours of the week, which distorts moving averages, ATR and inside bar patterns. New York close data gives five candles a week. If you backtest on one type and trade on a broker using the other, your daily signals won't match. Check the time of the last candle each day on your platform, and make sure your data source uses the same definition.

[Fairman]

Logging the source and version of each data set sounds fussy until you try to reproduce a result three months later and can't. Keep a short text file with each backtest: data provider, symbol, timeframe, date range, time zone, whether spreads were fixed or variable, and the date you downloaded it. Brokers sometimes update or fill gaps in history, so the same download a few months later can be slightly different. If a strategy result suddenly changes, you want to know whether the strategy or the data changed. Without that note, you can waste days chasing a difference that came from a different data feed.