ATLAS: Adapting Trajectory Lengths and Step-Size for Hamiltonian Monte Carlo
This addresses sampling inefficiencies for statisticians and machine learning practitioners dealing with complex posterior distributions, though it is an incremental improvement over existing adaptive methods.
The authors tackled the problem of Hamiltonian Monte Carlo (HMC) and No U-Turn Sampler (NUTS) struggling with complex geometric distributions by developing ATLAS, which locally adapts step size and trajectory length, resulting in accurate sampling for difficult distributions while being computationally competitive for simpler ones.
Hamiltonian Monte-Carlo (HMC) and its auto-tuned variant, the No U-Turn Sampler (NUTS) can struggle to accurately sample distributions with complex geometries, e.g., varying curvature, due to their constant step size for leapfrog integration and fixed mass matrix. In this work, we develop a strategy to locally adapt the step size parameter of HMC at every iteration by evaluating a low-rank approximation of the local Hessian and estimating its largest eigenvalue. We combine it with a strategy to similarly adapt the trajectory length by monitoring the no U-turn condition, resulting in an adaptive sampler, ATLAS: adapting trajectory length and step-size. We further use a delayed rejection framework for making multiple proposals that improves the computational efficiency of ATLAS, and develop an approach for automatically tuning its hyperparameters during warmup. We compare ATLAS with state-of-the-art samplers like NUTS on a suite of synthetic and real world examples, and show that i) unlike NUTS, ATLAS is able to accurately sample difficult distributions with complex geometries, ii) it is computationally competitive to NUTS for simpler distributions, and iii) it is more robust to the tuning of hyperparamters.