RECAPP: Crafting a More Efficient Catalyst for Convex Optimization
This work provides more efficient optimization algorithms for machine learning practitioners dealing with convex optimization problems, representing an incremental improvement over existing Catalyst methods.
The authors tackled the problem of extraneous logarithmic terms in convergence rates of accelerated proximal point algorithms by proposing RECAPP, a relaxed error criterion that eliminates the need for high-accuracy subproblem solutions. For finite-sum problems, they matched the best known complexity, and for max-structured minimization, they improved on the best Catalyst-based bound by a logarithmic factor.
The accelerated proximal point algorithm (APPA), also known as "Catalyst", is a well-established reduction from convex optimization to approximate proximal point computation (i.e., regularized minimization). This reduction is conceptually elegant and yields strong convergence rate guarantees. However, these rates feature an extraneous logarithmic term arising from the need to compute each proximal point to high accuracy. In this work, we propose a novel Relaxed Error Criterion for Accelerated Proximal Point (RECAPP) that eliminates the need for high accuracy subproblem solutions. We apply RECAPP to two canonical problems: finite-sum and max-structured minimization. For finite-sum problems, we match the best known complexity, previously obtained by carefully-designed problem-specific algorithms. For minimizing $\max_y f(x,y)$ where $f$ is convex in $x$ and strongly-concave in $y$, we improve on the best known (Catalyst-based) bound by a logarithmic factor.