Consistency Conditions for Differentiable Surrogate Losses
This work addresses a theoretical bottleneck for researchers designing consistent surrogate losses in machine learning, though it is incremental as it extends prior results from polyhedral to differentiable surrogates.
The paper tackles the problem of verifying statistical consistency for differentiable surrogate losses in discrete prediction tasks, showing that indirect elicitation (IE) is equivalent to calibration for one-dimensional convex differentiable losses but fails in higher dimensions, leading to the introduction of strong IE which ensures calibration for strongly convex, differentiable surrogates.
The statistical consistency of surrogate losses for discrete prediction tasks is often checked via the condition of calibration. However, directly verifying calibration can be arduous. Recent work shows that for polyhedral surrogates, a less arduous condition, indirect elicitation (IE), is still equivalent to calibration. We give the first results of this type for non-polyhedral surrogates, specifically the class of convex differentiable losses. We first prove that under mild conditions, IE and calibration are equivalent for one-dimensional losses in this class. We construct a counter-example that shows that this equivalence fails in higher dimensions. This motivates the introduction of strong IE, a strengthened form of IE that is equally easy to verify. We establish that strong IE implies calibration for differentiable surrogates and is both necessary and sufficient for strongly convex, differentiable surrogates. Finally, we apply these results to a range of problems to demonstrate the power of IE and strong IE for designing and analyzing consistent differentiable surrogates.