Resampling With Replacement vs Shuffling: Two Monte Carlo Methods Compared

[FXS Prop Desk]

Monte Carlo tests of trading results usually use one of two methods. They answer slightly different questions.

Shuffling (permutation): reorder the same trades randomly. The final result is always the same; only the path changes. It shows the range of drawdowns possible with exactly these trades.

Resampling with replacement (bootstrap): draw trades randomly, allowing repeats. Final results vary. It estimates the range of outcomes if the future resembles the past distribution.

Python example:

Code: Select all

import numpy as np
r = np.array(trades)
perm_dd = [np.minimum.accumulate(np.cumsum(np.random.permutation(r))).min() for _ in range(10000)]
boot_end = [np.random.choice(r, len(r)).sum() for _ in range(10000)]

Use shuffling for drawdown planning, bootstrap for estimating return uncertainty.

In both cases, the number of trades in the original sample matters. With fewer than about 100 trades, the simulation mostly reflects the randomness of the sample itself, and its results should be treated with caution.

Which method do you use, and what did it show about your strategy?

[Fairman]

Use shuffling for drawdown planning, bootstrap for estimating return uncertainty.

Agree with the split, but the shuffle line in the example measures something slightly different from drawdown. Taking the minimum of the cumulative sum gives the lowest point relative to the starting balance, so a path that runs up 20R and then drops 15R shows no drawdown at all. For peak-to-trough you need equity minus its running maximum, something like eq = np.cumsum(np.random.permutation(r)) and then dd = (eq - np.maximum.accumulate(eq)).min(). The original version is still useful for a static max-loss rule measured from the starting balance, but for trailing limits or for how much pain to expect, the peak-to-trough number is the one that matters.

[Shadow Trader]

FXS Prop Desk, I use both methods, but I lean on shuffling when sizing a soft stop and on bootstrap when I want a feel for return scatter. Fairman's note on peak to trough is the fix I needed after my first scripts understated pain.

With a path that runs up 12R and then gives back 8R, a start to trough measure barely blinks. Equity minus running max shows the 8R draw that actually hits a trailing floor.

I only trust either test once the sample clears about 120 trades. Below that the paths mostly replay sample noise, and I treat the percentiles as sketches rather than rules.

On my London EURUSD book the shuffle peak to trough 95th percentile sat near 6.5R over a 200 trade window. That is what set the soft stop, not the average equity curve.

I still run bootstrap on final P and L to see how often a quarter lands negative. Which percentile do you size from, and do you refresh the sample every month?

[Shadow Trader]

FXS Prop Desk, both methods share one assumption that bit me recently, which is that trades are independent. Shuffling single trades breaks up losing days, so if your losses cluster by session, the simulated drawdowns come out too gentle.

img_13711_r2.png (50.65 KiB) Viewed 54 times

The fix I use is shuffling whole days instead of single trades. On a 200 trade sample from my New York gold setup, five trades a day over 40 days, single trade shuffling put the 95th percentile peak to trough at 11.2R, while shuffling days as blocks put it at 13.8R.

The chart shows both distributions from 10,000 runs each on the same trades, so the final result is identical and only the clustering differs. My actual worst drawdown on that sample was 8.8R, close to the 9.0R day block median and well above the 7.0R single trade median.

A quick check for whether you need this is to compare your losing days against what independent trades would give. If red days arrive in pairs and runs more often than a coin would produce, size from the block version.

The same idea works with resampling, by drawing whole days with replacement. Do you see clustering in your own results, or do your losers arrive spread out?

[Shadow Trader]

A single backtest produces one sequence of trades, but the order in which trades occur is partly a matter of chance. Different sequences of the same trades yield different drawdowns, which matters for risk planning.

Monte Carlo resampling shuffles the order of trades, or draws trades randomly with replacement, thousands of times to produce a distribution of possible equity curves. From this distribution, estimates emerge for the likely range of maximum drawdown, the probability of hitting a loss threshold, and the variation in final results.

The method shows that the historical drawdown is only one sample from many possible outcomes, and that worse drawdowns than the backtest displayed are plausible. Position sizing decisions based on the ninety-fifth percentile drawdown, rather than the historical one, provide a safety margin. Simple tools in spreadsheets or short scripts can run the simulation. The technique cannot fix a flawed strategy, but it improves understanding of the uncertainty around a good one.

[FXS Prop Desk]

For peak-to-trough you need equity minus its running maximum

Right, the min of the cumulative sum is loss from the start, not peak-to-trough, and thanks for the corrected line. On a prop account both numbers matter and they map to different rules: the static max loss uses the original version, a trailing drawdown uses yours. Printing the percentiles of both from the same set of shuffles is worthwhile, because with a trailing limit the peak-to-trough figure is usually the one that binds first.

[Fairman]

Printing both from the same set of shuffles is a good habit, and it's cheap, since the equity series is already there. For the trailing limit I'd also record how often the peak-to-trough figure exceeds the buffer at each percentile, not just the percentile value.

A made-up run: 1,000 shuffles of 200 trades at 0.5 percent risk, average +0.15R. The start-to-trough 95th percentile loss might be 4.8 percent, while peak-to-trough at the 95th is 6.9 percent. If the firm's trailing limit is 6 percent, then the second number shows roughly 12 percent of paths breaching it, which the first one would have hidden entirely. That's the sort of gap that makes someone pass a challenge and lose the funded account a month later.

Since the peak-to-trough one binds first on a trailing rule, I'd size from it, and probably cut risk from 0.5 to 0.4 percent so the 95th percentile sits near 5.5 percent. It costs some speed in reaching the target but the odds of staying alive improve a lot.

One thing I'm unsure of is whether to use shuffling or resampling with replacement for that figure. Shuffling keeps the same total, so the end result is fixed, while resampling lets the total vary. Which do you use for the breach rate? And how many paths do you consider enough for the 95th percentile to settle?