Robust Stochastic Gradient Posterior Sampling with Lattice Based Discretisation
This addresses robustness issues in scalable Bayesian posterior sampling for practitioners, but it is incremental as it builds on existing lattice-based discretization methods.
The paper tackled the sensitivity of stochastic-gradient MCMC methods to minibatch size and gradient noise by proposing Stochastic Gradient Lattice Random Walk (SGLRW), which introduces noise only through off-diagonal covariance elements for robustness. Experimental results on Bayesian regression and classification showed SGLRW remains stable where SGLD fails, including with heavy-tailed noise, and matches or improves predictive performance.
Stochastic-gradient MCMC methods enable scalable Bayesian posterior sampling but often suffer from sensitivity to minibatch size and gradient noise. To address this, we propose Stochastic Gradient Lattice Random Walk (SGLRW), an extension of the Lattice Random Walk discretization. Unlike conventional Stochastic Gradient Langevin Dynamics (SGLD), SGLRW introduces stochastic noise only through the off-diagonal elements of the update covariance; this yields greater robustness to minibatch size while retaining asymptotic correctness. Furthermore, as comparison we analyze a natural analogue of SGLD utilizing gradient clipping. Experimental validation on Bayesian regression and classification demonstrates that SGLRW remains stable in regimes where SGLD fails, including in the presence of heavy-tailed gradient noise, and matches or improves predictive performance.