DIPPA: An improved Method for Bilinear Saddle Point Problems
This work addresses a computational bottleneck in optimization for machine learning, offering an incremental improvement by eliminating the need for proximal operators in certain scenarios.
The paper tackles bilinear saddle point problems where proximal operators are hard to compute, proposing a new gradient-only algorithm that achieves an optimal complexity bound with respect to the coupling condition number.
This paper studies bilinear saddle point problems $\min_{\bf{x}} \max_{\bf{y}} g(\bf{x}) + \bf{x}^{\top} \bf{A} \bf{y} - h(\bf{y})$, where the functions $g, h$ are smooth and strongly-convex. When the gradient and proximal oracle related to $g$ and $h$ are accessible, optimal algorithms have already been developed in the literature \cite{chambolle2011first, palaniappan2016stochastic}. However, the proximal operator is not always easy to compute, especially in constraint zero-sum matrix games \cite{zhang2020sparsified}. This work proposes a new algorithm which only requires the access to the gradients of $g, h$. Our algorithm achieves a complexity upper bound $\tilde{\mathcal{O}}\left( \frac{\|\bf{A}\|_2}{\sqrt{μ_x μ_y}} + \sqrt[4]{κ_x κ_y (κ_x + κ_y)} \right)$ which has optimal dependency on the coupling condition number $\frac{\|\bf{A}\|_2}{\sqrt{μ_x μ_y}}$ up to logarithmic factors.