Testing For Distribution Shifts with Conditional Conformal Test Martingales
It provides a more reliable and faster distribution-shift detector for sequential settings, addressing a known bottleneck in existing conformal test martingales.
The paper proposes a sequential test for detecting arbitrary distribution shifts that avoids test-time contamination by comparing each new sample to a fixed null reference dataset, achieving faster detection than standard conformal test martingales while maintaining anytime-valid type-I error control and asymptotic power one.
We propose a sequential test for detecting arbitrary distribution shifts that allows conformal test martingales (CTMs) to work under a fixed, reference-conditional setting. Existing CTM detectors construct test martingales by continually growing a reference set with each incoming sample, using it to assess how atypical the new sample is relative to past observations. While this design yields anytime-valid type-I error control, it suffers from test-time contamination: after a change, post-shift observations enter the reference set and dilute the evidence for distribution shift, increasing detection delay and reducing power. In contrast, our method avoids contamination by design by comparing each new sample to a fixed null reference dataset. Our main technical contribution is a robust martingale construction that remains valid conditional on the null reference data, achieved by explicitly accounting for the estimation error in the reference distribution induced by the finite reference set. This yields anytime-valid type-I error control together with guarantees of asymptotic power one and bounded expected detection delay. Empirically, our method detects shifts faster than standard CTMs, providing a powerful and reliable distribution-shift detector.