Optimal Computational Trade-Off of Inexact Proximal Methods
This work addresses optimization efficiency for machine learning practitioners, offering incremental improvements in computational strategies for proximal methods.
The paper tackles the trade-off between convergence rate and computational cost in proximal-gradient methods when the proximity operator is approximated iteratively, showing that a constant number of inner iterations minimizes computational cost for desired accuracy, and introduces SIP as a competitive alternative.
In this paper, we investigate the trade-off between convergence rate and computational cost when minimizing a composite functional with proximal-gradient methods, which are popular optimisation tools in machine learning. We consider the case when the proximity operator is computed via an iterative procedure, which provides an approximation of the exact proximity operator. In that case, we obtain algorithms with two nested loops. We show that the strategy that minimizes the computational cost to reach a solution with a desired accuracy in finite time is to set the number of inner iterations to a constant, which differs from the strategy indicated by a convergence rate analysis. In the process, we also present a new procedure called SIP (that is Speedy Inexact Proximal-gradient algorithm) that is both computationally efficient and easy to implement. Our numerical experiments confirm the theoretical findings and suggest that SIP can be a very competitive alternative to the standard procedure.