Neural Bootstrapper
This addresses a computational bottleneck for researchers and practitioners needing efficient uncertainty estimation in deep learning, though it is an incremental improvement on existing bootstrapping methods.
The paper tackles the computational burden of bootstrapping deep neural networks for uncertainty quantification by proposing Neural Bootstrapper (NeuBoots), which learns to generate bootstrapped networks through single training, reducing costs while outperforming other bagging methods in tasks like image classification and active learning.
Bootstrapping has been a primary tool for ensemble and uncertainty quantification in machine learning and statistics. However, due to its nature of multiple training and resampling, bootstrapping deep neural networks is computationally burdensome; hence it has difficulties in practical application to the uncertainty estimation and related tasks. To overcome this computational bottleneck, we propose a novel approach called \emph{Neural Bootstrapper} (NeuBoots), which learns to generate bootstrapped neural networks through single model training. NeuBoots injects the bootstrap weights into the high-level feature layers of the backbone network and outputs the bootstrapped predictions of the target, without additional parameters and the repetitive computations from scratch. We apply NeuBoots to various machine learning tasks related to uncertainty quantification, including prediction calibrations in image classification and semantic segmentation, active learning, and detection of out-of-distribution samples. Our empirical results show that NeuBoots outperforms other bagging based methods under a much lower computational cost without losing the validity of bootstrapping.