Analyzing Cost-Sensitive Surrogate Losses via $\mathcal{H}$-calibration
This addresses the practical problem of selecting optimal loss functions for classification tasks, particularly for small models, though it appears incremental as it builds on existing calibration theory.
This paper investigates whether cost-sensitive surrogate losses outperform cost-agnostic ones like cross-entropy for training machine learning models, finding that cost-sensitive surrogates strictly outperform under certain distributional assumptions and consistently beat cost-agnostic surrogates on UCI classification datasets.
This paper aims to understand whether machine learning models should be trained using cost-sensitive surrogates or cost-agnostic ones (e.g., cross-entropy). Analyzing this question through the lens of $\mathcal{H}$-calibration, we find that cost-sensitive surrogates can strictly outperform their cost-agnostic counterparts when learning small models under common distributional assumptions. Since these distributional assumptions are hard to verify in practice, we also show that cost-sensitive surrogates consistently outperform cost-agnostic surrogates on classification datasets from the UCI repository. Together, these make a strong case for using cost-sensitive surrogates in practice.