An Element-wise RSAV Algorithm for Unconstrained Optimization Problems
This work provides an incremental improvement in optimization algorithms for researchers in numerical analysis and machine learning, focusing on energy dissipation and convergence rates.
The authors tackled unconstrained optimization problems by introducing the element-wise relaxed scalar auxiliary variable (E-RSAV) algorithm, which ensures unconditional energy dissipation and improved energy alignment, with rigorous proofs of linear convergence in convex settings and super-linear convergence in univariate cases, validated by numerical experiments showing fast convergence.
We present a novel optimization algorithm, element-wise relaxed scalar auxiliary variable (E-RSAV), that satisfies an unconditional energy dissipation law and exhibits improved alignment between the modified and the original energy. Our algorithm features rigorous proofs of linear convergence in the convex setting. Furthermore, we present a simple accelerated algorithm that improves the linear convergence rate to super-linear in the univariate case. We also propose an adaptive version of E-RSAV with Steffensen step size. We validate the robustness and fast convergence of our algorithm through ample numerical experiments.