Probabilistic Conformal Coverage Guarantees in Small-Data Settings
This addresses the issue of unreliable risk control in practical applications of conformal prediction, particularly for small datasets, though it is an incremental improvement focused on a specific method.
The paper tackles the problem of high variance in realized coverage for split conformal prediction in small-data settings, introducing the Small Sample Beta Correction (SSBC) to provide probabilistic guarantees that ensure at least the desired coverage with user-defined probability.
Conformal prediction provides distribution-free prediction sets with guaranteed marginal coverage. However, in split conformal prediction this guarantee is training-conditional only in expectation: across many calibration draws, the average coverage equals the nominal level, but the realized coverage for a single calibration set may vary substantially. This variance undermines effective risk control in practical applications. Here we introduce the Small Sample Beta Correction (SSBC), a plug-and-play adjustment to the conformal significance level that leverages the exact finite-sample distribution of conformal coverage to provide probabilistic guarantees, ensuring that with user-defined probability over the calibration draw, the deployed predictor achieves at least the desired coverage.