Advertisement IC Markets

Making optimiser runs reproducible: data, settings and seeds

Share, develop, and backtest custom MQL4/MQL5 Expert Advisors, Python data-scraping scripts, trading bots, and automated market alert systems.
Post Reply
Fairman
Posts: 6955
Joined: Tue Jul 21, 2026 7:11 am
Location: Abuja

Making optimiser runs reproducible: data, settings and seeds

Post by Fairman »

Making optimiser runs reproducible: data, settings and seeds

Ever run an optimisation, got a nice result, then a month later tried to repeat it and got something different? That usually comes down to something changing that you didn't record.

The usual suspects. The history data changed, because the platform downloaded more or the broker's data got updated. The tester settings were different, like modelling mode, spread, or date range. The EA code changed slightly. Or the optimiser itself uses randomness, as genetic optimisation does, and you got a different path through the parameters.

What helps is boring record keeping. Save a copy of the data you tested on, or at least note its source and date range. Save the tester settings file alongside the results. Keep the EA version in version control and record the commit with each run. If your optimiser lets you set a random seed, set one and note it. MT5's genetic optimiser doesn't make that easy, so I'd at least run it a few times and compare.

The payoff is that when a result looks good, you can rerun it and check it's real. And when something looks off later, you can work out what changed.

Without that it's hard to tell whether a strategy improved or the test setup just drifted.
It’s Fairman :geek:
Recommended broker for automated trading & scalping IC Markets
Fairman
Posts: 6955
Joined: Tue Jul 21, 2026 7:11 am
Location: Abuja

Re: Making optimiser runs reproducible: data, settings and seeds

Post by Fairman »

Running the same genetic optimisation two or three times is a useful check in its own right. If the best parameters land in roughly the same area each time, the result is probably sturdy. If each run gives a completely different best set, the surface is noisy and the 'best' is likely luck. That ties into the plateau idea people talk about, where you want parameter regions that work broadly rather than one sharp peak that only one exact setting can reach. If you only run it once, you've got no way of telling which of those two situations you're in.
It’s Fairman :geek:
Fairman
Posts: 6955
Joined: Tue Jul 21, 2026 7:11 am
Location: Abuja

Re: Making optimiser runs reproducible: data, settings and seeds

Post by Fairman »

For Python based testing, pinning library versions matters more than people expect. A pandas or numpy update can change how some function handles edge cases, like resampling or rolling windows, and results shift slightly. A requirements file with exact versions, or a lock file, keeps that stable. It sounds like a software habit rather than a trading one, but backtests are software, and they break the same way. Same goes for your data cleaning scripts. If you change how you drop bad ticks or fill gaps, that's a new version of the test, and old results aren't directly comparable any more. A short note in the results folder saying what changed saves a lot of head scratching later.
It’s Fairman :geek:
Fairman
Posts: 6955
Joined: Tue Jul 21, 2026 7:11 am
Location: Abuja

Re: Making optimiser runs reproducible: data, settings and seeds

Post by Fairman »

Saving the full tester report and the settings file together for each run makes it much easier to come back later. A naming convention that includes the date, the symbol, the timeframe and the data source helps. Without it, after a few weeks of optimising, you end up with folders full of reports and no clear idea which one came from which settings or data. Even a simple text file listing each run is better than nothing. A small spreadsheet with one row per run, listing the key results, makes it easy to compare runs at a glance and spot when a result came from a different data set or tester version.
It’s Fairman :geek:
FXS Prop Desk
Posts: 1689
Joined: Wed Oct 07, 2026 9:52 am

Re: Making optimiser runs reproducible: data, settings and seeds

Post by FXS Prop Desk »

Recording the exact data range, the broker and the export date of the history is often the missing piece. Brokers sometimes revise historical data, and the same test on new data can give different results.

Saving the optimiser settings file together with the results is another simple step. In MT5, the set file and the report can be stored in the same folder with a date in the name, which makes it much easier to repeat a test months later.
FXS Official Team | The Idea Lab & Strategy
Fairman
Posts: 6955
Joined: Tue Jul 21, 2026 7:11 am
Location: Abuja

Re: Making optimiser runs reproducible: data, settings and seeds

Post by Fairman »

Different MT5 tester modes can give different results with the same settings. 'Every tick based on real ticks' uses the broker's tick history, while 'every tick' generates ticks from M1 bars. If the broker's tick data changes or you switch modes, results change. I'd always note which mode was used. For anything that trades frequently, the real ticks mode is closer to reality, but it depends on the broker having good tick history. Data from different brokers can also differ in tick density, so the same mode can give different results depending on whose history you downloaded. Noting the broker and the download date with each run helps explain differences later.
It’s Fairman :geek:
Post Reply