How to Test a Seasonal Claim Honestly
Posted: Wed Oct 28, 2026 2:16 am
Seasonal claims are easy to produce and hard to verify. A chart with an appealing pattern is not evidence, and a handful of steps separates a serious test from a story built after the fact.
The first step is to write the hypothesis before looking at data: which pair, which calendar window, which direction, and why. A reason rooted in a real flow or reporting date is stronger than one found by scanning. The second step is to count how many windows, pairs and months were examined, because the more that were tried, the more likely something looks significant by luck. A correction for multiple comparisons, or simply a stricter threshold, helps.
Next comes holding out data. The idea can be developed on one period and judged on another that was never used for tuning. Check whether the effect appears in related pairs, whether it survives removal of the single largest year, and whether the average is driven by a few outliers. Then include spreads, swaps and slippage, since a tiny average edge can disappear after costs.
Finally, look at stability over time. A pattern that fades in recent years is a warning. If an idea survives every check, it deserves cautious, small-scale forward testing, and if it fails, discarding it is a useful result rather than a loss.
The first step is to write the hypothesis before looking at data: which pair, which calendar window, which direction, and why. A reason rooted in a real flow or reporting date is stronger than one found by scanning. The second step is to count how many windows, pairs and months were examined, because the more that were tried, the more likely something looks significant by luck. A correction for multiple comparisons, or simply a stricter threshold, helps.
Next comes holding out data. The idea can be developed on one period and judged on another that was never used for tuning. Check whether the effect appears in related pairs, whether it survives removal of the single largest year, and whether the average is driven by a few outliers. Then include spreads, swaps and slippage, since a tiny average edge can disappear after costs.
Finally, look at stability over time. A pattern that fades in recent years is a warning. If an idea survives every check, it deserves cautious, small-scale forward testing, and if it fails, discarding it is a useful result rather than a loss.