Audited Conformal Prediction for Classification under Unknown Distribution Shift
For practitioners deploying pretrained models under unknown distribution shift, ACP provides a principled way to improve uncertainty quantification with minimal additional labeling.
Audited Conformal Prediction (ACP) uses a small labeled target dataset to train an audit model that identifies where a pretrained classifier fails, integrating this into conformal prediction to guarantee marginal coverage while achieving substantially higher conditional coverage than existing methods.
We consider the problem of uncertainty quantification for a pretrained classification model deployed under unknown distribution shift. We propose Audited Conformal Prediction (ACP), a method that leverages a small labeled dataset from the target population to train an auxiliary audit model identifying inputs where the legacy model is likely to fail. By integrating the audit model's outputs into the conformal prediction framework, ACP produces prediction sets that guarantee marginal coverage while achieving substantially higher conditional coverage in practice than existing approaches. We develop and analyze two complementary integration strategies -- one targeting marginal coverage with improved conditional performance, the other providing explicit group-conditional coverage guarantees -- and establish theoretical guarantees for both. Experiments on synthetic and real-world datasets validate the method and illustrate trade-offs between prediction set size and conditional coverage.