On optimal direction gibbs sampling
Provides a principled method for choosing directions in Gibbs sampling, improving MCMC efficiency for practitioners using gradient-free sampling.
The paper studies optimal direction selection for Directional Gibbs sampling, minimizing mutual information between MCMC steps for truncated Normal objectives, and generalizes to Multivariate Normal approximations. Tests on skewed non-normal functions show improved sampling efficiency.
Generalized Gibbs kernels are those that may take any direction not necessarily bounded to each axis along the parameters of the objective function. We study how to optimally choose such directions in a Directional, random scan, Gibbs sampler setting. The optimal direction is chosen by minimizing to the mutual information (Kullback-Leibler divergence) of two steps of the MCMC for a truncated Normal objective function. The result is generalized to be used when a Multivariate Normal (local) approximation is available for the objective function. Three Gibbs direction distributions are tested in highly skewed non-normal objective functions.