Uniform-in-time convergence bounds for Persistent Contrastive Divergence Algorithms
This work provides a novel method for training energy-based models with explicit error guarantees, addressing a foundational challenge in machine learning, though it appears incremental as it builds on existing PCD frameworks.
The authors tackled the problem of maximum likelihood estimation for unnormalized densities by proposing a continuous-time formulation of persistent contrastive divergence (PCD) as a coupled multiscale system of stochastic differential equations, which enabled them to derive explicit uniform-in-time error bounds for the PCD iterates relative to the MLE solution, with an efficient implementation using S-ROCK integrators providing explicit error guarantees.
We propose a continuous-time formulation of persistent contrastive divergence (PCD) for maximum likelihood estimation (MLE) of unnormalised densities. Our approach expresses PCD as a coupled, multiscale system of stochastic differential equations (SDEs), which perform optimisation of the parameter and sampling of the associated parametrised density, simultaneously. From this novel formulation, we are able to derive explicit bounds for the error between the PCD iterates and the MLE solution for the model parameter. This is made possible by deriving uniform-in-time (UiT) bounds for the difference in moments between the multiscale system and the averaged regime. An efficient implementation of the continuous-time scheme is introduced, leveraging a class of explicit, stable intregators, stochastic orthogonal Runge-Kutta Chebyshev (S-ROCK), for which we provide explicit error estimates in the long-time regime. This leads to a novel method for training energy-based models (EBMs) with explicit error guarantees.