Semi-Supervised Risk Control via Prediction-Powered Inference
This work addresses the sample-size barrier in risk control for machine learning practitioners, offering a method to reduce conservatism without compromising statistical validity, though it is incremental as it builds on existing frameworks.
The paper tackles the problem of noisy and conservative error rates in risk-controlling prediction sets due to limited labeled calibration data, by introducing a semi-supervised calibration procedure that leverages unlabeled data to tune hyper-parameters rigorously, resulting in improved performance demonstrated in few-shot image classification and early time series classification experiments.
The risk-controlling prediction sets (RCPS) framework is a general tool for transforming the output of any machine learning model to design a predictive rule with rigorous error rate control. The key idea behind this framework is to use labeled hold-out calibration data to tune a hyper-parameter that affects the error rate of the resulting prediction rule. However, the limitation of such a calibration scheme is that with limited hold-out data, the tuned hyper-parameter becomes noisy and leads to a prediction rule with an error rate that is often unnecessarily conservative. To overcome this sample-size barrier, we introduce a semi-supervised calibration procedure that leverages unlabeled data to rigorously tune the hyper-parameter without compromising statistical validity. Our procedure builds upon the prediction-powered inference framework, carefully tailoring it to risk-controlling tasks. We demonstrate the benefits and validity of our proposal through two real-data experiments: few-shot image classification and early time series classification.