Gaussian Switch Sampling: A Second Order Approach to Active Learning
This addresses active learning robustness for machine learning practitioners, though it is incremental as it builds on existing acquisition function methods.
The paper tackled the problem of active learning acquisition functions being sensitive to model changes and not robust to out-of-distribution settings by proposing a second-order definition of sample importance based on neural network forgetting, resulting in up to 5% improvement over existing strategies on various benchmarks.
In active learning, acquisition functions define informativeness directly on the representation position within the model manifold. However, for most machine learning models (in particular neural networks) this representation is not fixed due to the training pool fluctuations in between active learning rounds. Therefore, several popular strategies are sensitive to experiment parameters (e.g. architecture) and do not consider model robustness to out-of-distribution settings. To alleviate this issue, we propose a grounded second-order definition of information content and sample importance within the context of active learning. Specifically, we define importance by how often a neural network "forgets" a sample during training - artifacts of second order representation shifts. We show that our definition produces highly accurate importance scores even when the model representations are constrained by the lack of training data. Motivated by our analysis, we develop Gaussian Switch Sampling (GauSS). We show that GauSS is setup agnostic and robust to anomalous distributions with exhaustive experiments on three in-distribution benchmarks, three out-of-distribution benchmarks, and three different architectures. We report an improvement of up to 5% when compared against four popular query strategies.