Accelerating Multi-Block Constrained Optimization Through Learning to Optimize
This work addresses a critical bottleneck in multi-block optimization for researchers and practitioners, offering a method to improve convergence and performance in generic linearly constrained composite problems, though it is incremental as it builds on existing L2O and MPALM techniques.
The paper tackles the challenge of tuning penalty parameters in the Majorized Proximal Augmented Lagrangian Method (MPALM), a multi-block optimization method, by proposing a Learning to Optimize approach that adaptively selects hyperparameters using supervised learning. The results show that the framework outperforms popular alternatives in applications like the Lasso and optimal transport problems.
Learning to Optimize (L2O) approaches, including algorithm unrolling, plug-and-play methods, and hyperparameter learning, have garnered significant attention and have been successfully applied to the Alternating Direction Method of Multipliers (ADMM) and its variants. However, the natural extension of L2O to multi-block ADMM-type methods remains largely unexplored. Such an extension is critical, as multi-block methods leverage the separable structure of optimization problems, offering substantial reductions in per-iteration complexity. Given that classical multi-block ADMM does not guarantee convergence, the Majorized Proximal Augmented Lagrangian Method (MPALM), which shares a similar form with multi-block ADMM and ensures convergence, is more suitable in this setting. Despite its theoretical advantages, MPALM's performance is highly sensitive to the choice of penalty parameters. To address this limitation, we propose a novel L2O approach that adaptively selects this hyperparameter using supervised learning. We demonstrate the versatility and effectiveness of our method by applying it to the Lasso problem and the optimal transport problem. Our numerical results show that the proposed framework outperforms popular alternatives. Given its applicability to generic linearly constrained composite optimization problems, this work opens the door to a wide range of potential real-world applications.