Towards Practicable Sequential Shift Detectors
This work highlights critical gaps in making distribution shift detectors usable in real-world applications, which is an incremental step towards improving model reliability.
The paper addresses the problem of detecting distribution shifts in deployed machine learning models to prevent cost accumulation, but notes that existing sequential shift detectors lack key practical deployment requirements, and it identifies three such desiderata and suggests future research directions.
There is a growing awareness of the harmful effects of distribution shift on the performance of deployed machine learning models. Consequently, there is a growing interest in detecting these shifts before associated costs have time to accumulate. However, desiderata of crucial importance to the practicable deployment of sequential shift detectors are typically overlooked by existing works, precluding their widespread adoption. We identify three such desiderata, highlight existing works relevant to their satisfaction, and recommend impactful directions for future research.