Instance-Level Costs for Nuanced Classifier Evaluation
For practitioners in safety-critical applications, this work provides a practical evaluation metric but offers limited guidance on improving model training.
The paper introduces normalized excess cost (NEC), a metric that weights classification errors by per-example costs, and shows that models with 5% error rate can achieve 1.8% NEC, indicating most errors occur on ambiguous examples. However, cost-sensitive training yields inconsistent benefits, improving only when costs are predictable from input features.
Standard classification treats all errors equally, but in content moderation, medical screening, and safety-critical applications, mistakes on clear-cut cases are far more costly than errors on ambiguous ones. We propose normalized excess cost (NEC), a metric that weights classification errors by per-example costs and reduces to standard error rate when costs are uniform. Costs can derive from annotator vote margins, distance from decision thresholds, or confidence ratings. Across text, image, and tabular benchmarks, we find that NEC is often substantially lower than error rate -- models with 5\% error rate can achieve 1.8\% NEC -- revealing that most mistakes concentrate on ambiguous, low-cost examples. However, incorporating costs into training via loss weighting, sampling strategies, or regression yields inconsistent benefits: improvements appear only when costs are predictable from input features, as in our synthetic control, while real-world datasets show mixed or negligible gains. Our framework provides a practical methodology for deriving and evaluating instance-level misclassification costs, even when cost-sensitive training offers limited benefit.