Hyperbolic Optimization
This work addresses optimization challenges for learning on hyperbolic manifolds, particularly relevant for Poincaré embeddings, but it appears incremental as it builds directly on existing Riemannian optimization principles.
The authors tackled optimization on hyperbolic manifolds by extending Riemannian SGD to create Hyperbolic Adam, which accelerates early-stage convergence when parameters are far from the optimum. In a case study with diffusion models, they achieved faster convergence on certain datasets without sacrificing generative quality.
This work explores optimization methods on hyperbolic manifolds. Building on Riemannian optimization principles, we extend the Hyperbolic Stochastic Gradient Descent (a specialization of Riemannian SGD) to a Hyperbolic Adam optimizer. While these methods are particularly relevant for learning on the Poincaré ball, they may also provide benefits in Euclidean and other non-Euclidean settings, as the chosen optimization encourages the learning of Poincaré embeddings. This representation, in turn, accelerates convergence in the early stages of training, when parameters are far from the optimum. As a case study, we train diffusion models using the hyperbolic optimization methods with hyperbolic time-discretization of the Langevin dynamics, and show that they achieve faster convergence on certain datasets without sacrificing generative quality.