Read a Setup Scoreboard Without Fooling Yourself

[Fairman]

Read a Setup Scoreboard Without Fooling Yourself

A journal summary shows three setups. Setup A: 30 trades, total plus 12R. Setup B: 25 trades, total minus 4R. Setup C: 15 trades, total plus 9R.

Compute the averages. A: 12 divided by 30 is plus 0.40R per trade. B: minus 4 divided by 25 is minus 0.16R. C: 9 divided by 15 is plus 0.60R. Overall: 17R over 70 trades, plus 0.24R.

The tempting conclusions: drop B, and trade only C because its average is highest. Test each.

Sample size. Fifteen trades is small. With a standard deviation of 1.5R, the standard error for C is 1.5 divided by the square root of 15, which is 0.39R. Its average of 0.60 is about 1.5 standard errors above zero, promising but not proven. For B, the standard error is 0.30R, and its average of minus 0.16 is about half a standard error below zero, which is indistinguishable from breakeven.

So the data does not show that B is a losing setup. It shows that B has not yet proven itself.

What would you do? Keep collecting data on all three. Reduce risk on B if you must, or look at why B's trades differ. Do not scale up C on fifteen trades.

Also check whether the setups overlap in conditions. If C fires only in strong trends and B in ranges, the real variable may be the regime.

Your turn: compute the standard error for each setup in your own journal before ranking them.

[FXS HongKong Scalper]

Good to see the sample size issue highlighted. The difference between A and C is probably not significant with these numbers. Continuing to trade all three, perhaps with smaller size on B, until more data is collected seems more sensible than cutting setups too early.

[Fairman]

The standard error calculations are the key to reading the scoreboard honestly. Setup C's average of 0.60R looks best, but with only 15 trades and a standard error of 0.39R, the true average could easily be anywhere from about minus 0.2R to plus 1.4R. Setup B's minus 0.16R, with a standard error of 0.30R, is indistinguishable from zero. So the scoreboard tells us much less than the averages suggest. The tempting conclusions, dropping B and focusing on C, could be correct, but the data doesn't yet support them with any confidence. More trades are needed for all three setups.

[Fairman]

Setup A deserves attention as the most reliable result. With 30 trades and an average of 0.40R, its standard error is about 0.27R, so its average is about 1.5 standard errors above zero, similar to C. But A has twice as many trades, so its estimate is more stable. A trader looking for the most dependable setup might favour A over C, even though C's average is higher. In practice, the best approach is often to keep trading all three at normal risk until each has 50 or more trades, then reassess with tighter estimates.

[Amanhasnoname]

This post is close to something I went through with my own journal. I had a pullback setup that sat at minus 0.3R after 18 trades, and I dropped it, only to find months later in a backtest that it was fine in trending sessions and bad in the afternoon chop. I had thrown away a setup because of a sample that was too small and too mixed.

What I do now is tag every trade with the session and a simple trend or range label, then split each setup by that tag before judging it. Setup B in your example may be hiding the same thing, as the last reply hinted, a good setup in one regime and a bad one in the other. Splitting 25 trades in two gives tiny groups, so I treat the result as a question to explore, not an answer.

Meanwhile I cut risk on the doubtful setup to half and keep logging. I only review after 50 trades, never sooner.

[Fairman]

The point about regime overlap is important. If C fires only in strong trends and B only in ranges, then C's strong result and B's weak result might reflect the market conditions during the sample period, not the quality of the setups. If the market had been ranging more, B might look better and C worse. Tagging each trade with the market condition at entry, such as trending or ranging based on a simple rule, allows the results to be split by regime. That shows whether a setup works in its intended conditions, which is more useful than its overall average.

[Fairman]

Computing the standard error before ranking setups, as the post suggests, is a habit that prevents many premature decisions. The formula is simple: the standard deviation of the R results divided by the square root of the number of trades. A spreadsheet can calculate it automatically for each setup. Has anyone here dropped a setup after a poor run, only to find later that it had just been unlucky? I'd be interested in how many trades you had on it when you made the decision, and whether you later went back to it.