Test Distribution-Aware Active Learning: A Principled Approach Against Distribution Shift and Outliers
This addresses a fundamental shortfall in active learning for machine learning practitioners, but it is incremental as it builds on existing Bayesian methods.
The authors tackled the problem of conventional active learning methods failing to account for test-time distribution, which can lead to suboptimal acquisition strategies, and they proposed EPIG and JEPIG methods that showed improved performance under distribution shift, though no concrete numbers were provided.
Expanding on MacKay (1992), we argue that conventional model-based methods for active learning - like BALD - have a fundamental shortfall: they fail to directly account for the test-time distribution of the input variables. This can lead to pathologies in the acquisition strategy, as what is maximally informative for model parameters may not be maximally informative for prediction: for example, when the data in the pool set is more dispersed than that of the final prediction task, or when the distribution of pool and test samples differs. To correct this, we revisit an acquisition strategy that is based on maximizing the expected information gained about possible future predictions, referring to this as the Expected Predictive Information Gain (EPIG). As EPIG does not scale well for batch acquisition, we further examine an alternative strategy, a hybrid between BALD and EPIG, which we call the Joint Expected Predictive Information Gain (JEPIG). We consider using both for active learning with Bayesian neural networks on a variety of datasets, examining the behavior under distribution shift in the pool set.