Paper Splits Conformal Reliability From Frequency Calibration After a Break

A paper posted to arXiv by Columbia University researcher Karl Li proposes that conformal prediction, the technique that wraps a forecast in a band meant to contain the true outcome a set share of the time, needs two separate report cards once a system passes a known, announced boundary. One report card, the per-time audit, tracks how close the band’s coverage sits to a target miscoverage of 0.1 at each step. The other, the frequency audit, checks whether a closed window’s actual error rate matched that target over its whole span. The paper argues a single reset rule, built from the empirical quantile of scores observed after the boundary, can satisfy the first audit at close to the best possible rate for independent scores, and can keep the second audit’s error small and shrinking as more data arrives, provided a bounded floor is added to the recursion.

The author tests the reset rule against adaptive conformal methods run from the original authors’ published code, on a large synthetic break and on two real economic series: an electricity market reform and a set of monetary policy decisions. At the 50th forecast after the synthetic break, the reset rule’s gap against the nominal target beat every comparison method. On unseen scores following the market reform, its performance band sat below some comparison methods, tied others including adaptive conformal inference, and matched one exactly at zero.

The numbers

The paper reports announced boundaries tested in total, with one clearing the smallest coverage difference its own detection window could register. Around monetary policy decisions, the reset rule’s band sat above zero against every rival method on 10 of the first 34 days studied, a sign that real policy announcements are harder to reset around cleanly than the synthetic break or the electricity reform.

For an operator pricing a service by predicted load, the difference between a method that looks calibrated on average and a method that stays calibrated the moment a new boundary is announced is the difference between a smooth billing cycle and a week of refunds or throttling after every price change.