Testing on a different broker's feed as an out-of-sample check
Posted: Sat Oct 31, 2026 6:58 pm
Out-of-sample testing usually means a different time period. Another valuable test is a different data source. If a strategy works on one broker's feed and fails on another's, it may depend on that feed's peculiarities.
A robust edge should survive across reasonable variations in the data: different liquidity providers, slightly different spreads, different tick timing. Strong dependence on one feed suggests the strategy exploits artifacts, such as filtering or stale quotes.
Run the same logic on data from two or three sources for the same period. Compare the number of trades, the win rate, the average profit, and the cost assumptions.
Some difference is expected. Large discrepancies, such as a profit factor of 1.5 on one and 0.9 on another, are a warning.
Also consider the reverse. If a strategy works only on the broker where it will trade, check whether the edge comes from execution advantages such as price improvement or from artifacts that may vanish.
Cross-feed validation is cheap if the data is available, and it exposes fragile strategies early.
How different are the results across feeds for the same strategy and period?
A robust edge should survive across reasonable variations in the data: different liquidity providers, slightly different spreads, different tick timing. Strong dependence on one feed suggests the strategy exploits artifacts, such as filtering or stale quotes.
Run the same logic on data from two or three sources for the same period. Compare the number of trades, the win rate, the average profit, and the cost assumptions.
Some difference is expected. Large discrepancies, such as a profit factor of 1.5 on one and 0.9 on another, are a warning.
Also consider the reverse. If a strategy works only on the broker where it will trade, check whether the edge comes from execution advantages such as price improvement or from artifacts that may vanish.
Cross-feed validation is cheap if the data is available, and it exposes fragile strategies early.
How different are the results across feeds for the same strategy and period?