AutoCP: Automated Pipelines for Accurate Prediction Intervals
This addresses the need for reliable uncertainty quantification in real-world applications like financial forecasting and personalized medicine, representing an incremental improvement over existing conformal prediction methods.
The paper tackles the problem of constructing accurate and less conservative prediction intervals in machine learning, proposing AutoCP, an AutoML framework that optimizes interval length while achieving target coverage rates, and reports it significantly outperforms benchmark algorithms on various datasets.
Successful application of machine learning models to real-world prediction problems, e.g. financial forecasting and personalized medicine, has proved to be challenging, because such settings require limiting and quantifying the uncertainty in the model predictions, i.e. providing valid and accurate prediction intervals. Conformal Prediction is a distribution-free approach to construct valid prediction intervals in finite samples. However, the prediction intervals constructed by Conformal Prediction are often (because of over-fitting, inappropriate measures of nonconformity, or other issues) overly conservative and hence inadequate for the application(s) at hand. This paper proposes an AutoML framework called Automatic Machine Learning for Conformal Prediction (AutoCP). Unlike the familiar AutoML frameworks that attempt to select the best prediction model, AutoCP constructs prediction intervals that achieve the user-specified target coverage rate while optimizing the interval length to be accurate and less conservative. We tested AutoCP on a variety of datasets and found that it significantly outperforms benchmark algorithms.