Critique an Indicator That Claims Eighty Percent Accuracy

[Fairman]

Critique an Indicator That Claims Eighty Percent Accuracy

A vendor says its indicator "predicts reversals with 80 percent accuracy," based on a test of 25 signals, 20 of which were correct. Examine the claim.

Sample size. With 25 signals and 20 correct, the observed rate is 80 percent. The standard error is the square root of 0.8 times 0.2 divided by 25, which is 0.08. A 95 percent interval is 80 percent plus or minus 15.7 points: 64 to 96 percent. The true accuracy could be well below the advertised number.

Definition of accuracy. Accurate at what? A reversal might be defined as any move of 10 pips in the opposite direction within 50 candles, an easy standard in a volatile market. If reversals of that kind happen after 70 percent of random points, the indicator adds little. The base rate is the comparison that matters.

Payoff. Accuracy says nothing about how much is won or lost. An indicator that is right 80 percent of the time, with wins of 10 pips and losses of 60, loses money.

Repainting. Many indicators redraw past signals as new data arrives. On a historical chart, the signals appear perfectly placed, but in real time, they appeared later or changed. Test by saving a live screenshot and comparing the next day.

Out-of-sample. Was the test on data the creator used to design it?

Costs and conditions: did the test include spreads, and which pairs and times?

Verdict: the claim is unsupported. A fair test would use fixed rules, 100 or more signals, costs, and untouched data.

Exercise: write what you would need to see to believe an indicator's accuracy.

[FXS Prop Desk]

Fairman, the standard error calculation is exactly the right tool here. With 25 signals the interval around 80 percent is wide enough to include fairly ordinary accuracy. It is also worth asking what accuracy means for the vendor: a reversal predicted correctly but only after a large move against the trade is not much use for trading.

[Fairman]

The confidence interval calculation is a good first check. With 25 signals, the 95 percent interval runs from about 64 to 96 percent, which is very wide. The true accuracy could be anywhere in that range, and the vendor has chosen to report the point estimate as if it were precise. A larger sample would narrow it. With 250 signals and the same 80 percent rate, the interval would be about 75 to 85 percent. With 25, the claim is almost meaningless as a precise figure, even before considering the other problems the post identifies.

[Amanhasnoname]

I bought an indicator once that advertised something close to eighty percent, and I tested it the boring way before trusting it. I printed the chart, saved a screenshot at the end of each session for two weeks, and compared it with what the indicator showed the next morning. On the history it was perfect, but about a third of the arrows from the day before had either moved a few bars or vanished, so the live record looked quite different from the sales page.

What I did next was define my own rule for a correct signal, a ten pip move in my favour before a ten pip move against, and counted base rates on random bars. On EURUSD that simple rule was satisfied about 58 percent of the time with no indicator at all, which put the vendor's number in a very different light.

After that I only considered tools that let me log signals in real time with a timestamp, because that is the only record nobody can edit afterwards.

[Fairman]

The base rate point is the most important. If price moves 10 pips in the opposite direction after 70 percent of random entry points, an indicator that achieves 80 percent is only 10 points better than chance, and with 25 signals, that difference isn't statistically meaningful. Many indicator accuracy claims rely on generous definitions of success that random entries would meet most of the time. Asking how often a random signal would meet the same definition is the key question, and vendors rarely answer it. Testing it yourself on historical data is straightforward.

[Fairman]

Repainting is a serious and common problem. An indicator that uses future data to place past signals will look perfect on historical charts, because each signal is drawn after the outcome is known. In real time, the signal either appears later, after much of the move has happened, or appears and then disappears when price moves against it. The screenshot test suggested in the post is simple and effective: capture the chart with the signals at a given moment, then compare the same chart the next day. If signals have moved or vanished, the indicator repaints.

[Fairman]

The payoff point connects to the high win rate threads elsewhere on the forum. An 80 percent accuracy with wins of 10 pips and losses of 60 gives an expectancy of 0.8 times 10 minus 0.2 times 60, which is 8 minus 12, minus 4 pips per trade. Accuracy alone tells you nothing about profitability. Asking a vendor for the average win, average loss, and expectancy after costs is the fastest way to test the claim. Has anyone here bought an indicator with an accuracy claim and tested it independently? I'd be curious how its live performance compared with the advertised figure.