A Variational Analysis of Stochastic Gradient Algorithms
This provides a new method for optimizing hyperparameters in probabilistic models, connecting SGD to scalable inference algorithms, but it is incremental as it builds on existing variational inference concepts.
The paper tackles the problem of using Stochastic Gradient Descent (SGD) with constant learning rates as an approximate posterior inference algorithm for probabilistic modeling, showing that by adjusting tuning parameters, the stationary distribution of SGD can match the target posterior.
Stochastic Gradient Descent (SGD) is an important algorithm in machine learning. With constant learning rates, it is a stochastic process that, after an initial phase of convergence, generates samples from a stationary distribution. We show that SGD with constant rates can be effectively used as an approximate posterior inference algorithm for probabilistic modeling. Specifically, we show how to adjust the tuning parameters of SGD such as to match the resulting stationary distribution to the posterior. This analysis rests on interpreting SGD as a continuous-time stochastic process and then minimizing the Kullback-Leibler divergence between its stationary distribution and the target posterior. (This is in the spirit of variational inference.) In more detail, we model SGD as a multivariate Ornstein-Uhlenbeck process and then use properties of this process to derive the optimal parameters. This theoretical framework also connects SGD to modern scalable inference algorithms; we analyze the recently proposed stochastic gradient Fisher scoring under this perspective. We demonstrate that SGD with properly chosen constant rates gives a new way to optimize hyperparameters in probabilistic models.