On Measuring and Quantifying Performance: Error Rates, Surrogate Loss, and an Example in SSL
This addresses the problem of evaluating classifier performance in machine learning, particularly in semi-supervised learning, but is incremental as it builds on existing evaluation methods.
The paper argues that comparing classifiers based on surrogate loss, in addition to error rates, provides additional insights into performance, especially in semi-supervised learning where log-likelihood can guarantee improvements with more unlabeled data, unlike error rates.
In various approaches to learning, notably in domain adaptation, active learning, learning under covariate shift, semi-supervised learning, learning with concept drift, and the like, one often wants to compare a baseline classifier to one or more advanced (or at least different) strategies. In this chapter, we basically argue that if such classifiers, in their respective training phases, optimize a so-called surrogate loss that it may also be valuable to compare the behavior of this loss on the test set, next to the regular classification error rate. It can provide us with an additional view on the classifiers' relative performances that error rates cannot capture. As an example, limited but convincing empirical results demonstrates that we may be able to find semi-supervised learning strategies that can guarantee performance improvements with increasing numbers of unlabeled data in terms of log-likelihood. In contrast, the latter may be impossible to guarantee for the classification error rate.