Conformal inference is (almost) free for neural networks trained with early stopping
This work addresses the problem of achieving accurate and statistically reliable predictions in neural networks for practitioners, though it is incremental as it builds on existing early stopping and conformal inference techniques.
The paper tackles the lack of statistical guarantees in neural networks trained with early stopping by introducing conformalized early stopping, a method that combines early stopping with conformal calibration using the same hold-out data, resulting in models that provide exact predictive inferences without multiple data splits or conservative adjustments, as demonstrated on real data for tasks like outlier detection, classification, and regression.
Early stopping based on hold-out data is a popular regularization technique designed to mitigate overfitting and increase the predictive accuracy of neural networks. Models trained with early stopping often provide relatively accurate predictions, but they generally still lack precise statistical guarantees unless they are further calibrated using independent hold-out data. This paper addresses the above limitation with conformalized early stopping: a novel method that combines early stopping with conformal calibration while efficiently recycling the same hold-out data. This leads to models that are both accurate and able to provide exact predictive inferences without multiple data splits nor overly conservative adjustments. Practical implementations are developed for different learning tasks -- outlier detection, multi-class classification, regression -- and their competitive performance is demonstrated on real data.