Bilevel optimization for learning hyperparameters: Application to solving PDEs and inverse problems with Gaussian processes
This work addresses the problem of efficient hyperparameter optimization for researchers and practitioners in scientific computing, offering a scalable method for high-dimensional settings, though it is incremental as it builds on existing bilevel frameworks.
The paper tackles the computational challenge of hyperparameter tuning in bilevel optimization for PDEs and inverse problems by introducing a Gauss-Newton linearization strategy that reduces each outer iteration to a single linearized PDE solve, resulting in substantial improvements in accuracy and robustness compared to random initialization.
Methods for solving scientific computing and inference problems, such as kernel- and neural network-based approaches for partial differential equations (PDEs), inverse problems, and supervised learning tasks, depend crucially on the choice of hyperparameters. Specifically, the efficacy of such methods, and in particular their accuracy, stability, and generalization properties, strongly depends on the choice of hyperparameters. While bilevel optimization offers a principled framework for hyperparameter tuning, its nested optimization structure can be computationally demanding, especially in PDE-constrained contexts. In this paper, we propose an efficient strategy for hyperparameter optimization within the bilevel framework by employing a Gauss-Newton linearization of the inner optimization step. Our approach provides closed-form updates, eliminating the need for repeated costly PDE solves. As a result, each iteration of the outer loop reduces to a single linearized PDE solve, followed by explicit gradient-based hyperparameter updates. We demonstrate the effectiveness of the proposed method through Gaussian process models applied to nonlinear PDEs and to PDE inverse problems. Extensive numerical experiments highlight substantial improvements in accuracy and robustness compared to conventional random hyperparameter initialization. In particular, experiments with additive kernels and neural network-parameterized deep kernels demonstrate the method's scalability and effectiveness for high-dimensional hyperparameter optimization.