On Scaled Methods for Saddle Point Problems
This work addresses the optimization of saddle point problems in adversarial machine learning, such as GANs training, but is incremental as it analyzes existing scaling techniques without introducing new ones.
The paper conducted a theoretical analysis of adaptive scaling methods, including Adam, RmsProp, AdaHessian, and OASIS, for solving saddle point problems, with experimental studies on GANs showing good applicability across these methods.
Methods with adaptive scaling of different features play a key role in solving saddle point problems, primarily due to Adam's popularity for solving adversarial machine learning problems, including GANS training. This paper carries out a theoretical analysis of the following scaling techniques for solving SPPs: the well-known Adam and RmsProp scaling and the newer AdaHessian and OASIS based on Hutchison approximation. We use the Extra Gradient and its improved version with negative momentum as the basic method. Experimental studies on GANs show good applicability not only for Adam, but also for other less popular methods.