Bayesian Inference with Anchored Ensembles of Neural Networks, and Application to Exploration in Reinforcement Learning
This work provides a theoretical foundation for uncertainty estimation in neural networks, which is incremental but addresses a known bottleneck in reinforcement learning exploration.
The authors tackled the lack of analytical justification for using neural network ensembles in uncertainty quantification by proposing a minor modification that enables Bayesian inference, proving convergence to a Gaussian Process asymptotically. In reinforcement learning experiments, this technique led to steadier and more stable learning.
The use of ensembles of neural networks (NNs) for the quantification of predictive uncertainty is widespread. However, the current justification is intuitive rather than analytical. This work proposes one minor modification to the normal ensembling methodology, which we prove allows the ensemble to perform Bayesian inference, hence converging to the corresponding Gaussian Process as both the total number of NNs, and the size of each, tend to infinity. This working paper provides early-stage results in a reinforcement learning setting, analysing the practicality of the technique for an ensemble of small, finite number. Using the uncertainty estimates produced by anchored ensembles to govern the exploration-exploitation process results in steadier, more stable learning.