Stochastic Frank-Wolfe for Constrained Finite-Sum Minimization
This work addresses optimization challenges in machine learning for problems with constraints like sparsity or low-rank structures, offering a practical and efficient method, though it appears incremental as it builds on existing Stochastic Frank-Wolfe algorithms.
The paper tackles constrained smooth finite-sum minimization problems, such as empirical risk minimization with structured constraints, by proposing a novel Stochastic Frank-Wolfe algorithm that is simple to implement, requires no step-size tuning, and has constant per-iteration cost independent of dataset size, with benchmarks showing faster empirical convergence in some regimes.
We propose a novel Stochastic Frank-Wolfe (a.k.a. conditional gradient) algorithm for constrained smooth finite-sum minimization with a generalized linear prediction/structure. This class of problems includes empirical risk minimization with sparse, low-rank, or other structured constraints. The proposed method is simple to implement, does not require step-size tuning, and has a constant per-iteration cost that is independent of the dataset size. Furthermore, as a byproduct of the method we obtain a stochastic estimator of the Frank-Wolfe gap that can be used as a stopping criterion. Depending on the setting, the proposed method matches or improves on the best computational guarantees for Stochastic Frank-Wolfe algorithms. Benchmarks on several datasets highlight different regimes in which the proposed method exhibits a faster empirical convergence than related methods. Finally, we provide an implementation of all considered methods in an open-source package.