Primal-Dual Sequential Subspace Optimization for Saddle-point Problems
This incremental improvement addresses optimization challenges in scenarios like bi-linear games and GANs for researchers and practitioners in machine learning.
The paper tackles large-scale saddle-point problems by introducing a sequential subspace optimization method that uses low-dimensional subspaces derived from primal and dual first-order information, achieving significantly better convergence than popular first-order methods in experiments.
We introduce a new sequential subspace optimization method for large-scale saddle-point problems. It solves iteratively a sequence of auxiliary saddle-point problems in low-dimensional subspaces, spanned by directions derived from first-order information over the primal \emph{and} dual variables. Proximal regularization is further deployed to stabilize the optimization process. Experimental results demonstrate significantly better convergence relative to popular first-order methods. We analyze the influence of the subspace on the convergence of the algorithm, and assess its performance in various deterministic optimization scenarios, such as bi-linear games, ADMM-based constrained optimization and generative adversarial networks.