Stochastic DCA for minimizing a large sum of DC functions with application to Multi-class Logistic Regression
This work addresses optimization challenges in machine learning, particularly for multi-task learning applications, though it appears incremental as it builds on existing DCA methods with stochastic extensions.
The authors tackled the problem of minimizing large sums of DC functions, which arises in stochastic optimization and machine learning, by proposing stochastic DCA algorithms that guarantee convergence to critical points with probability one. They applied this to multi-class logistic regression for group variable selection, achieving superior classification accuracy, sparsity, and running time compared to existing methods on benchmark and synthetic datasets.
We consider the large sum of DC (Difference of Convex) functions minimization problem which appear in several different areas, especially in stochastic optimization and machine learning. Two DCA (DC Algorithm) based algorithms are proposed: stochastic DCA and inexact stochastic DCA. We prove that the convergence of both algorithms to a critical point is guaranteed with probability one. Furthermore, we develop our stochastic DCA for solving an important problem in multi-task learning, namely group variables selection in multi class logistic regression. The corresponding stochastic DCA is very inexpensive, all computations are explicit. Numerical experiments on several benchmark datasets and synthetic datasets illustrate the efficiency of our algorithms and their superiority over existing methods, with respect to classification accuracy, sparsity of solution as well as running time.