Resampling Stochastic Gradient Descent Cheaply for Efficient Uncertainty Quantification
This work addresses the need for efficient uncertainty quantification in SGD for machine learning practitioners, though it is incremental as it builds on existing bootstrap ideas.
The authors tackled the problem of uncertainty quantification for stochastic gradient descent (SGD) solutions by developing two computationally cheap resampling-based methods to construct confidence intervals, achieving this through enhancements of bootstrap schemes that reduce resampling requirements and bypass mixing conditions.
Stochastic gradient descent (SGD) or stochastic approximation has been widely used in model training and stochastic optimization. While there is a huge literature on analyzing its convergence, inference on the obtained solutions from SGD has only been recently studied, yet is important due to the growing need for uncertainty quantification. We investigate two computationally cheap resampling-based methods to construct confidence intervals for SGD solutions. One uses multiple, but few, SGDs in parallel via resampling with replacement from the data, and another operates this in an online fashion. Our methods can be regarded as enhancements of established bootstrap schemes to substantially reduce the computation effort in terms of resampling requirements, while at the same time bypassing the intricate mixing conditions in existing batching methods. We achieve these via a recent so-called cheap bootstrap idea and Berry-Esseen-type bound for SGD.