Don't Stop Me Yet: Sampling Loss Minima via Dissipative Riemannian Mechanics
For researchers in Bayesian deep learning, this provides a principled sampler for exploring loss minima, though the improvement is incremental over prior work.
The paper introduces DiMS, a sampler that exactly samples from the minimum level sets of neural network loss functions, which form connected components of reparameterization invariant solutions. It demonstrates improved uncertainty quantification in Bayesian inference compared to existing methods.
The minima of modern neural network loss functions are typically not isolated, rather they form connected components of reparameterization invariant solutions on the training data. Analytically characterizing these solutions is a hard problem, but sampling approaches are feasible. By construction, existing methods either spread over low-loss regions, and thus do not sample reparameterization invariant solutions exactly, or are inherently local, which limits exploration of other minima valleys. We propose sampling such reparameterization invariant models using a dynamical system based on kinetic energy, subject to a gravitational pull and a friction term that dissipates energy from the system. Our proposed sampler, DiMS, is guaranteed to sample exactly from the minimum level sets and depends on physically motivated hyperparameters which allows control over the exploration capabilities of the sampler. We consider uncertainty quantification in Bayesian inference as the motivating problem and observe improved performance compared to previously proposed approaches.