Riemannian Hamiltonian methods for min-max optimization on manifolds
This work addresses min-max optimization problems on manifolds, which is incremental as it extends Hamiltonian methods to Riemannian settings for applications in machine learning.
The paper tackles min-max optimization on Riemannian manifolds by introducing a Riemannian Hamiltonian function as a proxy, showing its minimizer corresponds to a saddle point under certain conditions, and proposes Riemannian Hamiltonian methods (RHM) with convergence analyses and extensions to stochastic settings, demonstrating efficacy in applications like subspace robust Wasserstein distance and GANs.
In this paper, we study min-max optimization problems on Riemannian manifolds. We introduce a Riemannian Hamiltonian function, minimization of which serves as a proxy for solving the original min-max problems. Under the Riemannian Polyak--Łojasiewicz condition on the Hamiltonian function, its minimizer corresponds to the desired min-max saddle point. We also provide cases where this condition is satisfied. For geodesic-bilinear optimization in particular, solving the proxy problem leads to the correct search direction towards global optimality, which becomes challenging with the min-max formulation. To minimize the Hamiltonian function, we propose Riemannian Hamiltonian methods (RHM) and present their convergence analyses. We extend RHM to include consensus regularization and to the stochastic setting. We illustrate the efficacy of the proposed RHM in applications such as subspace robust Wasserstein distance, robust training of neural networks, and generative adversarial networks.