Bilevel Learning with Inexact Stochastic Gradients
This work addresses the computational inefficiency of bilevel learning methods for machine learning and imaging practitioners, offering a more practical stochastic approach that is incremental over existing deterministic frameworks.
The paper tackled the problem of bilevel learning with inexact stochastic gradients, which is computationally challenging for large-scale applications like hyperparameter optimization and imaging, by establishing convergence under mild assumptions and demonstrating significant speed-ups and improved generalization in image denoising and deblurring tasks compared to deterministic methods.
Bilevel learning has gained prominence in machine learning, inverse problems, and imaging applications, including hyperparameter optimization, learning data-adaptive regularizers, and optimizing forward operators. The large-scale nature of these problems has led to the development of inexact and computationally efficient methods. Existing adaptive methods predominantly rely on deterministic formulations, while stochastic approaches often adopt a doubly-stochastic framework with impractical variance assumptions, enforces a fixed number of lower-level iterations, and requires extensive tuning. In this work, we focus on bilevel learning with strongly convex lower-level problems and a nonconvex sum-of-functions in the upper-level. Stochasticity arises from data sampling in the upper-level which leads to inexact stochastic hypergradients. We establish their connection to state-of-the-art stochastic optimization theory for nonconvex objectives. Furthermore, we prove the convergence of inexact stochastic bilevel optimization under mild assumptions. Our empirical results highlight significant speed-ups and improved generalization in imaging tasks such as image denoising and deblurring in comparison with adaptive deterministic bilevel methods.