OpReg-Boost: Learning to Accelerate Online Algorithms with Operator Regression
This work addresses the need for faster and more accurate online optimization in machine learning and control applications, representing an incremental improvement over prior techniques.
The paper tackles the problem of accelerating online optimization algorithms for time-varying convex composite costs by introducing OpReg-Boost, a regularization method that learns an algorithmic map for linear convergence, resulting in improved convergence and reduced asymptotic error compared to existing methods like FISTA and Anderson acceleration.
This paper presents a new regularization approach -- termed OpReg-Boost -- to boost the convergence and lessen the asymptotic error of online optimization and learning algorithms. In particular, the paper considers online algorithms for optimization problems with a time-varying (weakly) convex composite cost. For a given online algorithm, OpReg-Boost learns the closest algorithmic map that yields linear convergence; to this end, the learning procedure hinges on the concept of operator regression. We show how to formalize the operator regression problem and propose a computationally-efficient Peaceman-Rachford solver that exploits a closed-form solution of simple quadratically-constrained quadratic programs (QCQPs). Simulation results showcase the superior properties of OpReg-Boost w.r.t. the more classical forward-backward algorithm, FISTA, and Anderson acceleration.