Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score
For practitioners in high-stakes domains like medicine and finance, this provides a more reliable uncertainty quantification method that accounts for ordinal error severity.
The paper introduces a conformal prediction method for ordinal classification using the ranked probability score (RPS) as a nonconformity function, which yields median-centered contiguous prediction sets. Across multiple datasets, it achieves a favorable balance between prediction set width and ordinal miscoverage compared to existing methods.
Ordinal classification (OC) arises in high-stakes domains such as medicine and finance, where uncertainty quantification must account for the severity of ordinal errors. Conformal prediction (CP) provides distribution-free prediction sets with marginal coverage guarantees; however, its practical effectiveness depends critically on the choice of nonconformity function. We introduce a CP method for ordinal classification based on the ranked probability score (RPS), a proper scoring rule defined over cumulative predictive distributions. Although it reflects ordinal risk quite naturally, it has largely been neglected in conformal ordinal prediction (COP). When used as a measure of nonconformity, RPS yields median-centered contiguous prediction sets by construction. The method is model-agnostic, supports both assessed and grouped ordered categorical outcomes, and permits efficient implementation compared to greedy interval selection procedures. Across multiple ordinal image and tabular datasets, RPS-based CP produces contiguous prediction sets and strikes a favorable balance between prediction set width and the magnitude of ordinal miscoverage relative to existing CP methods.