Reducing Discretization Error in the Frank-Wolfe Method
This addresses a bottleneck in optimization for structurally constrained machine learning applications, offering incremental improvements to convergence stability.
The paper tackles the slow convergence and erratic step directions of the Frank-Wolfe algorithm by reducing discretization error, proposing a multistep method and an LMO-averaging scheme that accelerates the local convergence rate from O(1/k) to up to O(1/k^{3/2}).
The Frank-Wolfe algorithm is a popular method in structurally constrained machine learning applications, due to its fast per-iteration complexity. However, one major limitation of the method is a slow rate of convergence that is difficult to accelerate due to erratic, zig-zagging step directions, even asymptotically close to the solution. We view this as an artifact of discretization; that is to say, the Frank-Wolfe \emph{flow}, which is its trajectory at asymptotically small step sizes, does not zig-zag, and reducing discretization error will go hand-in-hand in producing a more stabilized method, with better convergence properties. We propose two improvements: a multistep Frank-Wolfe method that directly applies optimized higher-order discretization schemes; and an LMO-averaging scheme with reduced discretization error, and whose local convergence rate over general convex sets accelerates from a rate of $O(1/k)$ to up to $O(1/k^{3/2})$.