A Conditional Gradient-based Method for Simple Bilevel Optimization with Convex Lower-level Problem
This addresses a bottleneck in bilevel optimization for researchers and practitioners by providing improved convergence rates, though it is incremental as it builds on existing iterative methods.
The paper tackles the problem of simple bilevel optimization with a convex lower-level problem by introducing a method that uses cutting planes and conditional gradient updates, achieving iteration complexities of O(max{1/ε_f, 1/ε_g}) for convex upper-level objectives and O(max{1/ε_f^2, 1/(ε_f ε_g)}) for non-convex ones, which are claimed to be the best-known.
In this paper, we study a class of bilevel optimization problems, also known as simple bilevel optimization, where we minimize a smooth objective function over the optimal solution set of another convex constrained optimization problem. Several iterative methods have been developed for tackling this class of problems. Alas, their convergence guarantees are either asymptotic for the upper-level objective, or the convergence rates are slow and sub-optimal. To address this issue, in this paper, we introduce a novel bilevel optimization method that locally approximates the solution set of the lower-level problem via a cutting plane, and then runs a conditional gradient update to decrease the upper-level objective. When the upper-level objective is convex, we show that our method requires ${\mathcal{O}}(\max\{1/ε_f,1/ε_g\})$ iterations to find a solution that is $ε_f$-optimal for the upper-level objective and $ε_g$-optimal for the lower-level objective. Moreover, when the upper-level objective is non-convex, our method requires ${\mathcal{O}}(\max\{1/ε_f^2,1/(ε_fε_g)\})$ iterations to find an $(ε_f,ε_g)$-optimal solution. We also prove stronger convergence guarantees under the Hölderian error bound assumption on the lower-level problem. To the best of our knowledge, our method achieves the best-known iteration complexity for the considered class of bilevel problems.