Twin-Boot: Uncertainty-Aware Optimization via Online Two-Sample Bootstrapping
This addresses the need for uncertainty-aware optimization in machine learning, offering a novel method for integrating uncertainty estimation into training, though it is incremental in building on classical bootstrapping.
The paper tackled the problem of uncertainty estimation in deep learning, particularly in overparameterized and low-data regimes, by introducing Twin-Bootstrap Gradient Descent, which improved calibration and generalization in neural networks and inverse problems.
Standard gradient descent methods yield point estimates with no measure of confidence. This limitation is acute in overparameterized and low-data regimes, where models have many parameters relative to available data and can easily overfit. Bootstrapping is a classical statistical framework for uncertainty estimation based on resampling, but naively applying it to deep learning is impractical: it requires training many replicas, produces post-hoc estimates that cannot guide learning, and implicitly assumes comparable optima across runs - an assumption that fails in non-convex landscapes. We introduce Twin-Bootstrap Gradient Descent (Twin-Boot), a resampling-based training procedure that integrates uncertainty estimation into optimization. Two identical models are trained in parallel on independent bootstrap samples, and a periodic mean-reset keeps both trajectories in the same basin so that their divergence reflects local (within-basin) uncertainty. During training, we use this estimate to sample weights in an adaptive, data-driven way, providing regularization that favors flatter solutions. In deep neural networks and complex high-dimensional inverse problems, the approach improves calibration and generalization and yields interpretable uncertainty maps.