Optimization Guarantees for Square-Root Natural-Gradient Variational Inference
This work addresses a theoretical gap in variational inference for researchers, providing convergence guarantees that were previously challenging to establish, though it is incremental as it focuses on specific cases.
The paper tackled the lack of theoretical convergence guarantees for natural-gradient descent in variational inference, even for simple concave log-likelihood cases with Gaussian approximations, by introducing a square-root parameterization for the Gaussian covariance, which established novel convergence guarantees and demonstrated effectiveness in experiments with advantages over Euclidean or Wasserstein methods.
Variational inference with natural-gradient descent often shows fast convergence in practice, but its theoretical convergence guarantees have been challenging to establish. This is true even for the simplest cases that involve concave log-likelihoods and use a Gaussian approximation. We show that the challenge can be circumvented for such cases using a square-root parameterization for the Gaussian covariance. This approach establishes novel convergence guarantees for natural-gradient variational-Gaussian inference and its continuous-time gradient flow. Our experiments demonstrate the effectiveness of natural gradient methods and highlight their advantages over algorithms that use Euclidean or Wasserstein geometries.