Where to Get Historical Data and What to Watch For

[Fairman]

Where to Get Historical Data and What to Watch For

Testing requires data, and data quality affects the conclusions. Several sources exist, each with limits.

Brokers and platforms provide historical bar data, often for free, through their trading terminals. Quality varies. Gaps, missing periods, and differences between brokers appear. Data may be adjusted or modeled, especially for older periods.

Charting services like TradingView give access to historical data from various feeds, convenient for visual analysis, though depth of history and tick detail differ.

Third-party data vendors sell tick data and cleaner records, with better coverage and documented methods. Some offer free samples or limited free access, and prices rise with detail.

Exchange data, such as CME currency futures, comes with real volume and is good for studying market microstructure, though it covers futures only.

Central banks and statistical agencies provide free economic data and, in some cases, official exchange rates, useful for macro research.

Things to check include the time zone and the daily close definition, since these change candles. Whether the data includes bid, ask, or mid prices, and whether spread is recorded. The treatment of weekends and holidays. Adjustments for daylight saving.

Compare a sample against a second source to detect errors.

Keep a record of where each data set came from, and its version, so results can be reproduced.

Practical step: compare one week of hourly data for the same pair from two sources, and note any differences in highs and lows.

[Fairman]

The daily close definition is a quiet trap. Data based on New York 5pm and data based on midnight UTC give different daily candles, and therefore different signals. With a midnight UTC close, many brokers also show a small Sunday candle covering the first couple of hours of the week, which distorts moving averages, ATR and inside bar patterns. New York close data gives five candles a week. If you backtest on one type and trade on a broker using the other, your daily signals won't match. Check the time of the last candle each day on your platform, and make sure your data source uses the same definition.

[Fairman]

Logging the source and version of each data set sounds fussy until you try to reproduce a result three months later and can't. Keep a short text file with each backtest: data provider, symbol, timeframe, date range, time zone, whether spreads were fixed or variable, and the date you downloaded it. Brokers sometimes update or fill gaps in history, so the same download a few months later can be slightly different. If a strategy result suddenly changes, you want to know whether the strategy or the data changed. Without that note, you can waste days chasing a difference that came from a different data feed.

[Shadow Trader]

The Sunday candle problem is easy to underestimate until you see both versions. Take one made up week of EURUSD, built from the same hourly bars, split at 5pm New York in one row and at midnight UTC in the other.

img_13486_r1.png (79.53 KiB) Viewed 38 times

The chart is illustrative only: the New York split gives five candles, while the UTC split adds a Sunday stub just 12 pips tall. Average range per candle comes out at 71 pips on the New York version and 59 on the UTC version, and even Wednesday changes from 58 to 49 pips because its hours are cut differently.

For a scalper that gap flows into anything sized off daily range, such as a fractional stop or a quiet day filter. A strategy tuned on one split will see different quiet days on the other.

Bid versus mid is the other trap on short timeframes. If the data is bid only, a short's stop gets tested against the bid when it would really be hit on the ask, so every backtested short is stopped a spread later than it would be live.

Have you found a free source that stores the ask side as well? Most free downloads seem to be bid only, which is fine on H4 but not on M1.

[FXS Prop Desk]

Plot the data and look for flat periods, sudden jumps or missing weekends. These often reveal gaps or bad ticks. For tick data, comparing the number of ticks per day across the period also helps, since sudden drops usually mean incomplete data rather than a quiet market.

Using data from the same broker you trade with gives results closer to live conditions, even if the data is not perfect.

[Shadow Trader]

With data from your own broker, one catch is what the spread column stores. On MetaTrader the spread saved with each bar is typically the lowest spread seen during that bar, not the average, so an M1 test fed with it assumes the best moment of each minute.

Say a bar shows 0.3 pips while the spread spent most of that minute near 0.9. Across a scalping test with a 5 pip target, that 0.6 gap is 12 percent of the target on every trade, which is often the whole edge.

Tick data recorded live from the same broker fixes this, though it means waiting months for a usable sample. A middle route is to record spreads for a few weeks and apply the average for each hour of the day as a cost in the test.

Daylight saving also needs care with session filters. If the data is stored in UTC and the strategy trades the London open, the open sits an hour earlier in UTC during the British summer, so a fixed UTC window catches a different hour for half the year.

Does anyone add a fixed slippage figure on top of spread for stop exits in M1 tests? Half a pip is a common guess, though news bars need far more.

[Shadow Trader]

Several providers offer free historical tick data for major currency pairs, often as downloadable files or through a platform. For beginners and cost-conscious traders, they're a valuable resource. Their limits deserve attention.

Typical limitations include coverage of only the provider's own feed, which may differ from the broker where trades will be placed. Spreads in the data might not reflect the commission-based raw spreads of a particular account. Some data sets exclude certain hours, or fill gaps with synthetic ticks. Licence terms often restrict commercial use or redistribution.

Quality checks are a must. Look at the distribution of ticks per minute, check for gaps over weekends and holidays, scan for outliers in price and spread, and compare a sample against another source.

Despite the caveats, free data is a good starting point for exploring ideas, as long as results aren't treated as final. Anything promising should be re-tested on data from the intended broker, with realistic costs, before real money is committed.

Documenting the source, download date and cleaning steps makes results reproducible.

A habit that helps: keep a short note stating the source of every chart used for decisions, and the broker's equivalent symbol. When results diverge, the note makes the source of the mismatch obvious.

It's also worth checking time zone settings. Many charting services default to the user's local time zone or to exchange time, while the broker platform uses server time, so a candle that looks identical may actually be a different hour of the day.

Which free data sources have proved reliable, and what cleaning was necessary?