Loss-Controlling Calibration for Predictive Models
This work provides a more flexible calibration method for predictive models, addressing a specific technical bottleneck in loss control for researchers and practitioners in machine learning.
The authors tackled the problem of calibrating predictive models to control loss values for exchangeable data, extending conformal loss-controlling prediction to handle non-set predictors and non-monotone loss functions, with applications in selective regression and weather forecasting showing effectiveness.
We propose a learning framework for calibrating predictive models to make loss-controlling prediction for exchangeable data, which extends our recently proposed conformal loss-controlling prediction for more general cases. By comparison, the predictors built by the proposed loss-controlling approach are not limited to set predictors, and the loss function can be any measurable function without the monotone assumption. To control the loss values in an efficient way, we introduce transformations preserving exchangeability to prove finite-sample controlling guarantee when the test label is obtained, and then develop an approximation approach to construct predictors. The transformations can be built on any predefined function, which include using optimization algorithms for parameter searching. This approach is a natural extension of conformal loss-controlling prediction, since it can be reduced to the latter when the set predictors have the nesting property and the loss functions are monotone. Our proposed method is applied to selective regression and high-impact weather forecasting problems, which demonstrates its effectiveness for general loss-controlling prediction.