Banded Matrix Operators for Gaussian Markov Models in the Automatic Differentiation Era
This work addresses a computational bottleneck for researchers and practitioners using Gaussian Markov models, though it is incremental as it builds on existing automatic differentiation tools.
The paper tackles the challenge of enabling modern inference methods like variational inference and gradient-based sampling for Gaussian models with banded precision matrices, achieving this by integrating specialized linear algebra operators for banded matrices into automatic differentiation frameworks such as TensorFlow or PyTorch, with results showing linear complexity in the number of observations.
Banded matrices can be used as precision matrices in several models including linear state-space models, some Gaussian processes, and Gaussian Markov random fields. The aim of the paper is to make modern inference methods (such as variational inference or gradient-based sampling) available for Gaussian models with banded precision. We show that this can efficiently be achieved by equipping an automatic differentiation framework, such as TensorFlow or PyTorch, with some linear algebra operators dedicated to banded matrices. This paper studies the algorithmic aspects of the required operators, details their reverse-mode derivatives, and show that their complexity is linear in the number of observations.