Multivariate Gaussian Variational Inference by Natural Gradient Descent
Incremental improvement for variational inference practitioners.
The paper reviews Natural Gradient Descent for multivariate Gaussians, deriving the Fisher Information Matrix for various parameterizations and showing advantages of using mean and inverse covariance matrix with a simple update that accounts for symmetry and sparsity.
This short note reviews so-called Natural Gradient Descent (NGD) for multivariate Gaussians. The Fisher Information Matrix (FIM) is derived for several different parameterizations of Gaussians. Careful attention is paid to the symmetric nature of the covariance matrix when calculating derivatives. We show that there are some advantages to choosing a parameterization comprising the mean and inverse covariance matrix and provide a simple NGD update that accounts for the symmetric (and sparse) nature of the inverse covariance matrix.