A Hölderian backtracking method for min-max and min-min problems
This work addresses optimization challenges in machine learning, such as training GANs, but appears incremental as it builds on existing backtracking methods with specific regularity assumptions.
The authors tackled min-max and min-min optimization problems in non-convex settings by developing a Hölderian backtracking algorithm with automatic step size adaptation, achieving convergence guarantees and promising numerical results on GAN problems.
We present a new algorithm to solve min-max or min-min problems out of the convex world. We use rigidity assumptions, ubiquitous in learning, making our method applicable to many optimization problems. Our approach takes advantage of hidden regularity properties and allows us to devise a simple algorithm of ridge type. An original feature of our method is to come with automatic step size adaptation which departs from the usual overly cautious backtracking methods. In a general framework, we provide convergence theoretical guarantees and rates. We apply our findings on simple GAN problems obtaining promising numerical results.