Multinomial Logistic Regression Algorithms via Quadratic Gradient
This work provides incremental improvements to optimization algorithms for multiclass logistic regression, a fundamental classification method in machine learning.
The authors extended a recently proposed quadratic gradient technique from binary to multinomial logistic regression, developing enhanced versions of Nesterov's accelerated gradient and Adagrad that converge faster on multiclass datasets.
Multinomial logistic regression, also known by other names such as multiclass logistic regression and softmax regression, is a fundamental classification method that generalizes binary logistic regression to multiclass problems. A recently work proposed a faster gradient called $\texttt{quadratic gradient}$ that can accelerate the binary logistic regression training, and presented an enhanced Nesterov's accelerated gradient (NAG) method for binary logistic regression. In this paper, we extend this work to multiclass logistic regression and propose an enhanced Adaptive Gradient Algorithm (Adagrad) that can accelerate the original Adagrad method. We test the enhanced NAG method and the enhanced Adagrad method on some multiclass-problem datasets. Experimental results show that both enhanced methods converge faster than their original ones respectively.