Fast Proximal Linearized Alternating Direction Method of Multiplier with Parallel Splitting
This work provides incremental improvements to optimization methods like PALM and ADMM, benefiting researchers and practitioners in machine learning and related fields dealing with convex programming.
The authors tackled the problem of slow convergence in convex optimization with smooth and nonsmooth objective parts by proposing Fast PALM and Fast PL-ADMM-PS methods, achieving a convergence rate of O(1/K^2) compared to the traditional O(1/K) and showing significant improvements in experiments on synthetic and real-world data.
The Augmented Lagragian Method (ALM) and Alternating Direction Method of Multiplier (ADMM) have been powerful optimization methods for general convex programming subject to linear constraint. We consider the convex problem whose objective consists of a smooth part and a nonsmooth but simple part. We propose the Fast Proximal Augmented Lagragian Method (Fast PALM) which achieves the convergence rate $O(1/K^2)$, compared with $O(1/K)$ by the traditional PALM. In order to further reduce the per-iteration complexity and handle the multi-blocks problem, we propose the Fast Proximal ADMM with Parallel Splitting (Fast PL-ADMM-PS) method. It also partially improves the rate related to the smooth part of the objective function. Experimental results on both synthesized and real world data demonstrate that our fast methods significantly improve the previous PALM and ADMM.