A Positive Test Result and the Base Rate Problem
Posted: Thu Oct 29, 2026 10:28 am
A Positive Test Result and the Base Rate Problem
Scenario: suppose 10 percent of strategies people test truly have an edge. A backtest detects a real edge 80 percent of the time, and wrongly flags a no-edge strategy as profitable 5 percent of the time. A strategy passes the test. How likely is it that it truly has an edge?
Use counts. Out of 1,000 strategies: 100 have an edge, 900 do not. The test passes 80 percent of the 100, which is 80. It passes 5 percent of the 900, which is 45. Total passes: 125.
The share of passes that are real: 80 divided by 125, which is 64 percent.
So even after passing a test with 80 percent power and a 5 percent false-positive rate, the chance that the strategy is real is only 64 percent, since true edges are rare.
Now suppose edges are rarer, 2 percent of strategies. Out of 1,000: 20 real, 980 not. The test passes 16 of the real and 49 of the others. Total 65. The real share: 16 divided by 65, about 25 percent.
At a 2 percent base rate, a passing strategy is wrong three times out of four.
Why does this matter in trading? Most ideas fail, and markets are competitive, so the base rate of real edges is low. A good test result should raise your confidence, but it does not settle the matter.
The remedy is more evidence: out-of-sample tests, forward testing, and a plausible reason.
Exercise: compute the real share for a base rate of 5 percent and a test with a 1 percent false positive rate.
Scenario: suppose 10 percent of strategies people test truly have an edge. A backtest detects a real edge 80 percent of the time, and wrongly flags a no-edge strategy as profitable 5 percent of the time. A strategy passes the test. How likely is it that it truly has an edge?
Use counts. Out of 1,000 strategies: 100 have an edge, 900 do not. The test passes 80 percent of the 100, which is 80. It passes 5 percent of the 900, which is 45. Total passes: 125.
The share of passes that are real: 80 divided by 125, which is 64 percent.
So even after passing a test with 80 percent power and a 5 percent false-positive rate, the chance that the strategy is real is only 64 percent, since true edges are rare.
Now suppose edges are rarer, 2 percent of strategies. Out of 1,000: 20 real, 980 not. The test passes 16 of the real and 49 of the others. Total 65. The real share: 16 divided by 65, about 25 percent.
At a 2 percent base rate, a passing strategy is wrong three times out of four.
Why does this matter in trading? Most ideas fail, and markets are competitive, so the base rate of real edges is low. A good test result should raise your confidence, but it does not settle the matter.
The remedy is more evidence: out-of-sample tests, forward testing, and a plausible reason.
Exercise: compute the real share for a base rate of 5 percent and a test with a 1 percent false positive rate.