An In-Depth Examination of Risk Assessment in Multi-Class Classification Algorithms
This addresses the need for reliable risk assessment in safety-critical domains like healthcare and engineering, though it appears incremental by comparing existing and new methods.
The paper tackled the problem of estimating the probability of misclassification in multi-class classification algorithms for safety-critical applications, comparing calibration techniques and a novel conformal prediction-based approach, with the latter providing reasonable results across various models and datasets.
Advanced classification algorithms are being increasingly used in safety-critical applications like health-care, engineering, etc. In such applications, miss-classifications made by ML algorithms can result in substantial financial or health-related losses. To better anticipate and prepare for such losses, the algorithm user seeks an estimate for the probability that the algorithm miss-classifies a sample. We refer to this task as the risk-assessment. For a variety of models and datasets, we numerically analyze the performance of different methods in solving the risk-assessment problem. We consider two solution strategies: a) calibration techniques that calibrate the output probabilities of classification models to provide accurate probability outputs; and b) a novel approach based upon the prediction interval generation technique of conformal prediction. Our conformal prediction based approach is model and data-distribution agnostic, simple to implement, and provides reasonable results for a variety of use-cases. We compare the different methods on a broad variety of models and datasets.