The Certainty Ratio $C_ρ$: a novel metric for assessing the reliability of classifier predictions
This addresses the need for more reliable classifier evaluation in high-stakes applications, offering a tool for researchers and practitioners, though it is incremental as it builds on existing probabilistic frameworks.
The paper tackles the problem of evaluating classifier reliability by introducing the Certainty Ratio (C_ρ), a novel metric that quantifies the contribution of confident versus uncertain predictions to performance measures, and experimental results across 21 datasets show it reveals insights overlooked by traditional metrics.
Evaluating the performance of classifiers is critical in machine learning, particularly in high-stakes applications where the reliability of predictions can significantly impact decision-making. Traditional performance measures, such as accuracy and F-score, often fail to account for the uncertainty inherent in classifier predictions, leading to potentially misleading assessments. This paper introduces the Certainty Ratio ($C_ρ$), a novel metric designed to quantify the contribution of confident (certain) versus uncertain predictions to any classification performance measure. By integrating the Probabilistic Confusion Matrix ($CM^\star$) and decomposing predictions into certainty and uncertainty components, $C_ρ$ provides a more comprehensive evaluation of classifier reliability. Experimental results across 21 datasets and multiple classifiers, including Decision Trees, Naive-Bayes, 3-Nearest Neighbors, and Random Forests, demonstrate that $C_ρ$ reveals critical insights that conventional metrics often overlook. These findings emphasize the importance of incorporating probabilistic information into classifier evaluation, offering a robust tool for researchers and practitioners seeking to improve model trustworthiness in complex environments.