Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shifts
This addresses the challenge of detecting harmful distribution shifts for deployed models in real-world applications, representing an incremental improvement in risk monitoring techniques.
The paper tackles the problem of monitoring model performance in dynamic environments with limited labeled data by proposing prediction-powered risk monitoring (PPRM), which constructs anytime-valid lower bounds on running risk using synthetic and true labels, and demonstrates effectiveness in experiments on image classification, LLM, and telecommunications tasks.
We study the problem of monitoring model performance in dynamic environments where labeled data are limited. To this end, we propose prediction-powered risk monitoring (PPRM), a semi-supervised risk-monitoring approach based on prediction-powered inference (PPI). PPRM constructs anytime-valid lower bounds on the running risk by combining synthetic labels with a small set of true labels. Harmful shifts are detected via a threshold-based comparison with an upper bound on the nominal risk, satisfying assumption-free finite-sample guarantees in the probability of false alarm. We demonstrate the effectiveness of PPRM through extensive experiments on image classification, large language model (LLM), and telecommunications monitoring tasks.