Towards optimally abstaining from prediction with OOD test examples
This work addresses the challenge of distribution shifts in test data for machine learning practitioners, offering a method to improve reliability by abstaining from uncertain predictions, though it builds incrementally on prior research.
The paper tackles the problem of out-of-distribution (OOD) test examples in machine learning by proposing a transductive abstention algorithm that allows models to abstain from prediction at a fixed cost, with optimal loss guarantees that include an additional term based on distributional distance or adversarial fraction. For linear regression, it provides a polynomial-time algorithm, and for binary classification, it shows efficient implementation using an agnostic learner.
A common challenge across all areas of machine learning is that training data is not distributed like test data, due to natural shifts, "blind spots," or adversarial examples; such test examples are referred to as out-of-distribution (OOD) test examples. We consider a model where one may abstain from predicting, at a fixed cost. In particular, our transductive abstention algorithm takes labeled training examples and unlabeled test examples as input, and provides predictions with optimal prediction loss guarantees. The loss bounds match standard generalization bounds when test examples are i.i.d. from the training distribution, but add an additional term that is the cost of abstaining times the statistical distance between the train and test distribution (or the fraction of adversarial examples). For linear regression, we give a polynomial-time algorithm based on Celis-Dennis-Tapia optimization algorithms. For binary classification, we show how to efficiently implement it using a proper agnostic learner (i.e., an Empirical Risk Minimizer) for the class of interest. Our work builds on a recent abstention algorithm of Goldwasser, Kalais, and Montasser (2020) for transductive binary classification.