Criterion Collapse and Loss Distribution Control
This addresses a theoretical problem in machine learning optimization, providing insights into when different criteria converge, which is incremental as it builds on prior work on CVaR and DRO.
The paper tackles the problem of 'criterion collapse,' where optimizing one metric leads to optimality in another, specifically for error probability minimizers under various learning criteria like DRO, OCE risks, and non-monotonic methods. It shows that collapse extends beyond existing results for CVaR and DRO, and identifies conditions where monotonic criteria cannot avoid collapse while non-monotonic alternatives can.
In this work, we consider the notion of "criterion collapse," in which optimization of one metric implies optimality in another, with a particular focus on conditions for collapse into error probability minimizers under a wide variety of learning criteria, ranging from DRO and OCE risks (CVaR, tilted ERM) to non-monotonic criteria underlying recent ascent-descent algorithms explored in the literature (Flooding, SoftAD). We show how collapse in the context of losses with a Bernoulli distribution goes far beyond existing results for CVaR and DRO, then expand our scope to include surrogate losses, showing conditions where monotonic criteria such as tilted ERM cannot avoid collapse, whereas non-monotonic alternatives can.