Functional Central Limit Theorem and Strong Law of Large Numbers for Stochastic Gradient Langevin Dynamics
This provides theoretical guarantees for SGLD in practical settings with dependent data, addressing a gap for researchers in optimization and statistical learning.
The paper tackles the mixing properties of stochastic gradient Langevin dynamics (SGLD) with fixed step size under non-independent data streams, deriving a strong law of large numbers and a functional central limit theorem as key results.
We study the mixing properties of an important optimization algorithm of machine learning: the stochastic gradient Langevin dynamics (SGLD) with a fixed step size. The data stream is not assumed to be independent hence the SGLD is not a Markov chain, merely a \emph{Markov chain in a random environment}, which complicates the mathematical treatment considerably. We derive a strong law of large numbers and a functional central limit theorem for SGLD.