A Latent Variational Framework for Stochastic Optimization
This work provides a foundational theoretical framework for researchers in optimization and machine learning, though it is incremental as it unifies existing methods rather than introducing new algorithms.
The paper tackles the lack of a unifying theoretical framework for stochastic optimization algorithms by proposing a latent variational framework that connects them to Forward Backward Stochastic Differential Equations and Bayesian inference on gradients, recovering existing adaptive methods like stochastic gradient descent.
This paper provides a unifying theoretical framework for stochastic optimization algorithms by means of a latent stochastic variational problem. Using techniques from stochastic control, the solution to the variational problem is shown to be equivalent to that of a Forward Backward Stochastic Differential Equation (FBSDE). By solving these equations, we recover a variety of existing adaptive stochastic gradient descent methods. This framework establishes a direct connection between stochastic optimization algorithms and a secondary Bayesian inference problem on gradients, where a prior measure on noisy gradient observations determines the resulting algorithm.