Deeper Connections between Neural Networks and Gaussian Processes Speed-up Active Learning
This work addresses computational bottlenecks in active learning for researchers and practitioners, though it is incremental as it builds on existing methods.
The paper tackled the inefficiency and overconfidence issues in active learning for neural networks by approximating Bayesian neural networks with Gaussian processes, enabling efficient uncertainty updates without retraining and improving performance on real-world chemical and physical modeling tasks.
Active learning methods for neural networks are usually based on greedy criteria which ultimately give a single new design point for the evaluation. Such an approach requires either some heuristics to sample a batch of design points at one active learning iteration, or retraining the neural network after adding each data point, which is computationally inefficient. Moreover, uncertainty estimates for neural networks sometimes are overconfident for the points lying far from the training sample. In this work we propose to approximate Bayesian neural networks (BNN) by Gaussian processes, which allows us to update the uncertainty estimates of predictions efficiently without retraining the neural network, while avoiding overconfident uncertainty prediction for out-of-sample points. In a series of experiments on real-world data including large-scale problems of chemical and physical modeling, we show superiority of the proposed approach over the state-of-the-art methods.