A Comparison of First-order Algorithms for Machine Learning
This work provides a comparative analysis for researchers and practitioners in machine learning, but it is incremental as it focuses on existing algorithms.
The paper compared state-of-the-art first-order optimization algorithms for convex machine learning problems, finding that primal-dual algorithms performed best in terms of ease of construction, running time, and accuracy.
Using an optimization algorithm to solve a machine learning problem is one of mainstreams in the field of science. In this work, we demonstrate a comprehensive comparison of some state-of-the-art first-order optimization algorithms for convex optimization problems in machine learning. We concentrate on several smooth and non-smooth machine learning problems with a loss function plus a regularizer. The overall experimental results show the superiority of primal-dual algorithms in solving a machine learning problem from the perspectives of the ease to construct, running time and accuracy.