A Field Guide to Forward-Backward Splitting with a FASTA Implementation
For practitioners in machine learning and signal processing, this review and the FASTA solver offer practical guidance and a tool for solving optimization problems, but the work is incremental as it reviews existing methods.
This article reviews forward-backward splitting for non-differentiable and constrained optimization, focusing on practical implementation issues such as stepsize selection, acceleration, and stopping conditions. It introduces FASTA, a solver that implements these variations and provides a simple interface for a broad range of problems.
Non-differentiable and constrained optimization play a key role in machine learning, signal and image processing, communications, and beyond. For high-dimensional minimization problems involving large datasets or many unknowns, the forward-backward splitting method provides a simple, practical solver. Despite its apparently simplicity, the performance of the forward-backward splitting is highly sensitive to implementation details. This article is an introductory review of forward-backward splitting with a special emphasis on practical implementation concerns. Issues like stepsize selection, acceleration, stopping conditions, and initialization are considered. Numerical experiments are used to compare the effectiveness of different approaches. Many variations of forward-backward splitting are implemented in the solver FASTA (short for Fast Adaptive Shrinkage/Thresholding Algorithm). FASTA provides a simple interface for applying forward-backward splitting to a broad range of problems.