How long to judge a system after a change

[Amanhasnoname]

A tweak gets made, three days pass, and the results look worse. Back it goes. Or the results look better and the change is declared a success. Neither conclusion is justified by that little data.

Short-term trading generates many trades, but the noise is large. To tell a change of 0.2 pips per trade from random fluctuation takes hundreds of trades.

A practical rule is deciding in advance how many trades the test needs, and sticking to it. Without that, it is too easy to stop when the numbers look good or quit when they look bad.

Another useful tool is comparing the distribution of results, not just the average. A change that improves the average by increasing a few big winners might not be a real improvement.

Running the old and new versions in parallel, if possible, controls for market conditions.

Patience is a real edge here. Many traders are stuck in an endless cycle of tinkering because every conclusion comes too early.

How many trades go into judging a change, and was the number set in advance?