A First-Order Multi-Gradient Algorithm for Multi-Objective Bi-Level Optimization
This work addresses computational bottlenecks in MOBLO for researchers and practitioners in optimization and machine learning, offering an incremental improvement over prior methods.
The paper tackles the computational inefficiency of existing gradient-based methods for Multi-Objective Bi-Level Optimization (MOBLO) by proposing FORUM, a first-order multi-gradient algorithm that avoids Hessian matrix computations. It achieves state-of-the-art performance on three multi-task learning benchmark datasets, demonstrating effectiveness and efficiency in experiments.
In this paper, we study the Multi-Objective Bi-Level Optimization (MOBLO) problem, where the upper-level subproblem is a multi-objective optimization problem and the lower-level subproblem is for scalar optimization. Existing gradient-based MOBLO algorithms need to compute the Hessian matrix, causing the computational inefficient problem. To address this, we propose an efficient first-order multi-gradient method for MOBLO, called FORUM. Specifically, we reformulate MOBLO problems as a constrained multi-objective optimization (MOO) problem via the value-function approach. Then we propose a novel multi-gradient aggregation method to solve the challenging constrained MOO problem. Theoretically, we provide the complexity analysis to show the efficiency of the proposed method and a non-asymptotic convergence result. Empirically, extensive experiments demonstrate the effectiveness and efficiency of the proposed FORUM method in different learning problems. In particular, it achieves state-of-the-art performance on three multi-task learning benchmark datasets. The code is available at https://github.com/Baijiong-Lin/FORUM.