Fast Riemannian-manifold Hamiltonian Monte Carlo for hierarchical Gaussian-process models
This work addresses the need for effective Monte Carlo algorithms for analyzing real-world data with Gaussian processes, though it is incremental as it builds on existing RMHMC methods.
The study tackled the slow inference problem in hierarchical Gaussian-process models by optimizing Riemannian-manifold Hamiltonian Monte Carlo (RMHMC) with computation order optimization and dynamic programming of eigendecomposition, achieving drastically improved performance compared to existing libraries, as demonstrated numerically on simulated and real-world data.
Hierarchical Bayesian models based on Gaussian processes are considered useful for describing complex nonlinear statistical dependencies among variables in real-world data. However, effective Monte Carlo algorithms for inference with these models have not yet been established, except for several simple cases. In this study, we show that, compared with the slow inference achieved with existing program libraries, the performance of Riemannian-manifold Hamiltonian Monte Carlo (RMHMC) can be drastically improved by optimising the computation order according to the model structure and dynamically programming the eigendecomposition. This improvement cannot be achieved when using an existing library based on a naive automatic differentiator. We numerically demonstrate that RMHMC effectively samples from the posterior, allowing the calculation of model evidence, in a Bayesian logistic regression on simulated data and in the estimation of propensity functions for the American national medical expenditure data using several Bayesian multiple-kernel models. These results lay a foundation for implementing effective Monte Carlo algorithms for analysing real-world data with Gaussian processes, and highlight the need to develop a customisable library set that allows users to incorporate dynamically programmed objects and finely optimises the mode of automatic differentiation depending on the model structure.