Riemannian Bilevel Optimization
This work addresses optimization problems on Riemannian manifolds for researchers in machine learning and optimization, offering more efficient and implementable methods, though it is incremental as it builds on existing bilevel optimization techniques.
The paper tackles Riemannian bilevel optimization by developing new first-order gradient-based algorithms, specifically RF^2SA, which avoids second-order information and provides explicit convergence rates for reaching ε-stationary points in batch and stochastic setups.
We develop new algorithms for Riemannian bilevel optimization. We focus in particular on batch and stochastic gradient-based methods, with the explicit goal of avoiding second-order information such as Riemannian hyper-gradients. We propose and analyze $\mathrm{RF^2SA}$, a method that leverages first-order gradient information to navigate the complex geometry of Riemannian manifolds efficiently. Notably, $\mathrm{RF^2SA}$ is a single-loop algorithm, and thus easier to implement and use. Under various setups, including stochastic optimization, we provide explicit convergence rates for reaching $ε$-stationary points. We also address the challenge of optimizing over Riemannian manifolds with constraints by adjusting the multiplier in the Lagrangian, ensuring convergence to the desired solution without requiring access to second-order derivatives.