ASPEST: Bridging the Gap Between Active Learning and Selective Prediction
This work addresses the challenge of unreliable predictions under domain shift for applications requiring human-in-the-loop systems, offering a novel integration of two previously separate approaches.
The paper tackles the problem of domain shift in machine learning by introducing a new paradigm called active selective prediction, which combines active learning and selective prediction to query informative samples from shifted target domains, resulting in improved accuracy and coverage, such as increasing AUACC from 79.36% to 88.84% on the MNIST→SVHN benchmark with a labeling budget of 100.
Selective prediction aims to learn a reliable model that abstains from making predictions when uncertain. These predictions can then be deferred to humans for further evaluation. As an everlasting challenge for machine learning, in many real-world scenarios, the distribution of test data is different from the training data. This results in more inaccurate predictions, and often increased dependence on humans, which can be difficult and expensive. Active learning aims to lower the overall labeling effort, and hence human dependence, by querying the most informative examples. Selective prediction and active learning have been approached from different angles, with the connection between them missing. In this work, we introduce a new learning paradigm, active selective prediction, which aims to query more informative samples from the shifted target domain while increasing accuracy and coverage. For this new paradigm, we propose a simple yet effective approach, ASPEST, that utilizes ensembles of model snapshots with self-training with their aggregated outputs as pseudo labels. Extensive experiments on numerous image, text and structured datasets, which suffer from domain shifts, demonstrate that ASPEST can significantly outperform prior work on selective prediction and active learning (e.g. on the MNIST$\to$SVHN benchmark with the labeling budget of 100, ASPEST improves the AUACC metric from 79.36% to 88.84%) and achieves more optimal utilization of humans in the loop.