An Adaptively Inexact Method for Bilevel Learning Using Primal-Dual Style Differentiation
This work addresses the problem of efficient optimization in bilevel learning for machine learning practitioners, though it is incremental as it builds on existing primal-dual differentiation methods.
The paper tackles the challenge of bilevel learning for linear operators by proposing an adaptively inexact method that estimates hypergradient accuracy and adapts step sizes, achieving competitive performance on learned regularizer problems like training input-convex neural networks.
We consider a bilevel learning framework for learning linear operators. In this framework, the learnable parameters are optimized via a loss function that also depends on the minimizer of a convex optimization problem (denoted lower-level problem). We utilize an iterative algorithm called `piggyback' to compute the gradient of the loss and minimizer of the lower-level problem. Given that the lower-level problem is solved numerically, the loss function and thus its gradient can only be computed inexactly. To estimate the accuracy of the computed hypergradient, we derive an a-posteriori error bound, which provides guides for setting the tolerance for the lower-level problem, as well as the piggyback algorithm. To efficiently solve the upper-level optimization, we also propose an adaptive method for choosing a suitable step-size. To illustrate the proposed method, we consider a few learned regularizer problems, such as training an input-convex neural network.