Variational Optimization on Lie Groups, with Examples of Leading (Generalized) Eigenvalue Problems
This work addresses optimization problems on Lie groups, such as leading generalized eigenvalue problems, for researchers in mathematical optimization and machine learning, but it is incremental as it extends existing vector space methods to a new geometric setting.
The paper tackles smooth optimization on Lie groups by generalizing the NAG variational principle to these manifolds, resulting in continuous dynamics that converge to local optima and can be discretized into structure-preserving algorithms. Numerical experiments on synthetic data and LDA for MNIST show the methods are effective as optimization algorithms, though no specific performance numbers are provided.
The article considers smooth optimization of functions on Lie groups. By generalizing NAG variational principle in vector space (Wibisono et al., 2016) to Lie groups, continuous Lie-NAG dynamics which are guaranteed to converge to local optimum are obtained. They correspond to momentum versions of gradient flow on Lie groups. A particular case of $\mathsf{SO}(n)$ is then studied in details, with objective functions corresponding to leading Generalized EigenValue problems: the Lie-NAG dynamics are first made explicit in coordinates, and then discretized in structure preserving fashions, resulting in optimization algorithms with faithful energy behavior (due to conformal symplecticity) and exactly remaining on the Lie group. Stochastic gradient versions are also investigated. Numerical experiments on both synthetic data and practical problem (LDA for MNIST) demonstrate the effectiveness of the proposed methods as optimization algorithms ($not$ as a classification method).