Validity, consonant plausibility measures, and conformal prediction
This work addresses a foundational gap in uncertainty quantification for predictive distributions, relevant to statisticians and machine learning practitioners, though it appears incremental by extending existing conformal prediction frameworks.
The paper tackles the problem of quantifying prediction uncertainty by introducing a new notion of validity (Type-2) for predictive distributions, which complements the existing Type-1 validity for prediction regions, and shows that both types can be achieved by interpreting conformal prediction outputs as consonant plausibility measures, with proofs of validity provided.
Prediction of future observations is an important and challenging problem. The two mainstream approaches for quantifying prediction uncertainty use prediction regions and predictive distributions, respectively, with the latter believed to be more informative because it can perform other prediction-related tasks. The standard notion of validity, what we refer to here as Type-1 validity, focuses on coverage probability of prediction regions, while a notion of validity relevant to the other prediction-related tasks performed by predictive distributions is lacking. Here we present a new notion, called Type-2 validity, relevant to these other prediction tasks. We establish connections between Type-2 validity and coherence properties, and show that imprecise probability considerations are required in order to achieve it. We go on to show that both types of prediction validity can be achieved by interpreting the conformal prediction output as the contour function of a consonant plausibility measure. We also offer an alternative characterization of conformal prediction, based on a new nonparametric inferential model construction, wherein the appearance of consonance is natural, and prove its validity.