Making optimiser runs reproducible: data, settings and seeds

[Fairman]

Making optimiser runs reproducible: data, settings and seeds

Ever run an optimisation, got a nice result, then a month later tried to repeat it and got something different? That usually comes down to something changing that you didn't record.

The usual suspects. The history data changed, because the platform downloaded more or the broker's data got updated. The tester settings were different, like modelling mode, spread, or date range. The EA code changed slightly. Or the optimiser itself uses randomness, as genetic optimisation does, and you got a different path through the parameters.

What helps is boring record keeping. Save a copy of the data you tested on, or at least note its source and date range. Save the tester settings file alongside the results. Keep the EA version in version control and record the commit with each run. If your optimiser lets you set a random seed, set one and note it. MT5's genetic optimiser doesn't make that easy, so I'd at least run it a few times and compare.

The payoff is that when a result looks good, you can rerun it and check it's real. And when something looks off later, you can work out what changed.

Without that it's hard to tell whether a strategy improved or the test setup just drifted.

[Fairman]

Running the same genetic optimisation two or three times is a useful check in its own right. If the best parameters land in roughly the same area each time, the result is probably sturdy. If each run gives a completely different best set, the surface is noisy and the 'best' is likely luck. That ties into the plateau idea people talk about, where you want parameter regions that work broadly rather than one sharp peak that only one exact setting can reach. If you only run it once, you've got no way of telling which of those two situations you're in.

[Fairman]

For Python based testing, pinning library versions matters more than people expect. A pandas or numpy update can change how some function handles edge cases, like resampling or rolling windows, and results shift slightly. A requirements file with exact versions, or a lock file, keeps that stable. It sounds like a software habit rather than a trading one, but backtests are software, and they break the same way. Same goes for your data cleaning scripts. If you change how you drop bad ticks or fill gaps, that's a new version of the test, and old results aren't directly comparable any more. A short note in the results folder saying what changed saves a lot of head scratching later.