Generalization and Informativeness of Conformal Prediction
This work addresses the problem of improving uncertainty quantification for decision-making in machine learning by providing theoretical insights into conformal prediction, though it is incremental as it builds on existing CP methods.
The paper tackles the lack of control over the informativeness (average size) of conformal prediction sets by establishing a theoretical connection between the generalization properties of the base predictor and the informativeness of the resulting sets, deriving an upper bound on expected set size and validating it with numerical experiments.
The safe integration of machine learning modules in decision-making processes hinges on their ability to quantify uncertainty. A popular technique to achieve this goal is conformal prediction (CP), which transforms an arbitrary base predictor into a set predictor with coverage guarantees. While CP certifies the predicted set to contain the target quantity with a user-defined tolerance, it does not provide control over the average size of the predicted sets, i.e., over the informativeness of the prediction. In this work, a theoretical connection is established between the generalization properties of the base predictor and the informativeness of the resulting CP prediction sets. To this end, an upper bound is derived on the expected size of the CP set predictor that builds on generalization error bounds for the base predictor. The derived upper bound provides insights into the dependence of the average size of the CP set predictor on the amount of calibration data, the target reliability, and the generalization performance of the base predictor. The theoretical insights are validated using simple numerical regression and classification tasks.