Handling the Positive-Definite Constraint in the Bayesian Learning Rule
This work addresses a constraint-handling problem for researchers applying Bayesian learning rules in parameter spaces, representing an incremental improvement.
The paper tackles the issue of the Bayesian learning rule not satisfying positive-definite constraints in variational inference, proposing an improved rule using Riemannian gradient methods that outperforms existing methods without significant computational increase.
The Bayesian learning rule is a natural-gradient variational inference method, which not only contains many existing learning algorithms as special cases but also enables the design of new algorithms. Unfortunately, when variational parameters lie in an open constraint set, the rule may not satisfy the constraint and requires line-searches which could slow down the algorithm. In this work, we address this issue for positive-definite constraints by proposing an improved rule that naturally handles the constraints. Our modification is obtained by using Riemannian gradient methods, and is valid when the approximation attains a \emph{block-coordinate natural parameterization} (e.g., Gaussian distributions and their mixtures). We propose a principled way to derive Riemannian gradients and retractions from scratch. Our method outperforms existing methods without any significant increase in computation. Our work makes it easier to apply the rule in the presence of positive-definite constraints in parameter spaces.