Statistical Inference for the Population Landscape via Moment Adjusted Stochastic Gradients
This addresses the need for uncertainty quantification in statistical inference tasks using iterative optimization, which is crucial for statisticians and data scientists, though it appears incremental as it builds on existing stochastic gradient methods.
The paper tackles the problem of providing statistical inference and uncertainty quantification for solutions obtained via iterative optimization methods by introducing moment-adjusted stochastic gradient descents, establishing non-asymptotic theory that allows for model mis-specification and applies to both convex and non-convex cases, with numerical experiments showing acceleration effects.
Modern statistical inference tasks often require iterative optimization methods to compute the solution. Convergence analysis from an optimization viewpoint only informs us how well the solution is approximated numerically but overlooks the sampling nature of the data. In contrast, recognizing the randomness in the data, statisticians are keen to provide uncertainty quantification, or confidence, for the solution obtained using iterative optimization methods. This paper makes progress along this direction by introducing the moment-adjusted stochastic gradient descents, a new stochastic optimization method for statistical inference. We establish non-asymptotic theory that characterizes the statistical distribution for certain iterative methods with optimization guarantees. On the statistical front, the theory allows for model mis-specification, with very mild conditions on the data. For optimization, the theory is flexible for both convex and non-convex cases. Remarkably, the moment-adjusting idea motivated from "error standardization" in statistics achieves a similar effect as acceleration in first-order optimization methods used to fit generalized linear models. We also demonstrate this acceleration effect in the non-convex setting through numerical experiments.