Statistical inference using SGD
This method addresses the need for efficient statistical inference in large-scale problems, offering a first-order solution that is incremental in improving computational efficiency over existing techniques.
The paper tackles the problem of performing frequentist statistical inference in M-estimation problems by proposing a novel method based on averaging stochastic gradient descent (SGD) sequences with a fixed step size, demonstrating that this approach achieves accuracy comparable to classical methods while potentially requiring less computation.
We present a novel method for frequentist statistical inference in $M$-estimation problems, based on stochastic gradient descent (SGD) with a fixed step size: we demonstrate that the average of such SGD sequences can be used for statistical inference, after proper scaling. An intuitive analysis using the Ornstein-Uhlenbeck process suggests that such averages are asymptotically normal. From a practical perspective, our SGD-based inference procedure is a first order method, and is well-suited for large scale problems. To show its merits, we apply it to both synthetic and real datasets, and demonstrate that its accuracy is comparable to classical statistical methods, while requiring potentially far less computation.