Training Over-parameterized Models with Non-decomposable Objectives
This addresses the challenge of optimizing complex objectives in machine learning applications, offering a solution for training over-parameterized models more effectively, though it is incremental as it builds on existing cost-sensitive learning techniques.
The paper tackles the problem of training over-parameterized models with non-decomposable objectives, such as minimizing worst-case error or enforcing group-fairness constraints, by proposing new cost-sensitive losses that extend logit adjustment to handle general cost matrices, resulting in improved performance on benchmark image datasets with ResNet models.
Many modern machine learning applications come with complex and nuanced design goals such as minimizing the worst-case error, satisfying a given precision or recall target, or enforcing group-fairness constraints. Popular techniques for optimizing such non-decomposable objectives reduce the problem into a sequence of cost-sensitive learning tasks, each of which is then solved by re-weighting the training loss with example-specific costs. We point out that the standard approach of re-weighting the loss to incorporate label costs can produce unsatisfactory results when used to train over-parameterized models. As a remedy, we propose new cost-sensitive losses that extend the classical idea of logit adjustment to handle more general cost matrices. Our losses are calibrated, and can be further improved with distilled labels from a teacher model. Through experiments on benchmark image datasets, we showcase the effectiveness of our approach in training ResNet models with common robust and constrained optimization objectives.