A Framework and Benchmark for Deep Batch Active Learning for Regression
This work addresses sample efficiency in neural network regression for practitioners dealing with large datasets, though it is incremental as it builds on existing Bayesian and non-Bayesian methods.
The authors tackled the problem of expensive label acquisition in supervised learning by developing a framework for deep batch active learning in regression, which outperformed state-of-the-art methods on a benchmark of 15 large tabular data sets with improved scalability and ease of use.
The acquisition of labels for supervised learning can be expensive. To improve the sample efficiency of neural network regression, we study active learning methods that adaptively select batches of unlabeled data for labeling. We present a framework for constructing such methods out of (network-dependent) base kernels, kernel transformations, and selection methods. Our framework encompasses many existing Bayesian methods based on Gaussian process approximations of neural networks as well as non-Bayesian methods. Additionally, we propose to replace the commonly used last-layer features with sketched finite-width neural tangent kernels and to combine them with a novel clustering method. To evaluate different methods, we introduce an open-source benchmark consisting of 15 large tabular regression data sets. Our proposed method outperforms the state-of-the-art on our benchmark, scales to large data sets, and works out-of-the-box without adjusting the network architecture or training code. We provide open-source code that includes efficient implementations of all kernels, kernel transformations, and selection methods, and can be used for reproducing our results.