Quantifying Learning Guarantees for Convex but Inconsistent Surrogates
This work addresses the theoretical understanding of inconsistent surrogates for researchers in machine learning optimization, offering incremental extensions to existing frameworks.
The paper tackles the problem of quantifying learning guarantees for convex but inconsistent surrogates in machine learning, extending a framework to analyze inconsistency and deriving a non-trivial lower bound on the calibration function for the quadratic surrogate, which shows how inconsistent surrogates can provide guarantees on sample complexity and optimization difficulty.
We study consistency properties of machine learning methods based on minimizing convex surrogates. We extend the recent framework of Osokin et al. (2017) for the quantitative analysis of consistency properties to the case of inconsistent surrogates. Our key technical contribution consists in a new lower bound on the calibration function for the quadratic surrogate, which is non-trivial (not always zero) for inconsistent cases. The new bound allows to quantify the level of inconsistency of the setting and shows how learning with inconsistent surrogates can have guarantees on sample complexity and optimization difficulty. We apply our theory to two concrete cases: multi-class classification with the tree-structured loss and ranking with the mean average precision loss. The results show the approximation-computation trade-offs caused by inconsistent surrogates and their potential benefits.