Hybrid Coordinate Descent for Efficient Neural Network Learning Using Line Search and Gradient Descent
This work addresses computational efficiency in neural network optimization, but it is incremental as it builds on existing coordinate descent and gradient methods with a specific hybrid approach.
The paper tackles the problem of efficient neural network training by introducing a hybrid coordinate descent algorithm that combines line search and gradient descent for parameter updates, achieving improved computational efficiency through parallelization and threshold-based switching.
This paper presents a novel coordinate descent algorithm leveraging a combination of one-directional line search and gradient information for parameter updates for a squared error loss function. Each parameter undergoes updates determined by either the line search or gradient method, contingent upon whether the modulus of the gradient of the loss with respect to that parameter surpasses a predefined threshold. Notably, a larger threshold value enhances algorithmic efficiency. Despite the potentially slower nature of the line search method relative to gradient descent, its parallelizability facilitates computational time reduction. Experimental validation conducted on a 2-layer Rectified Linear Unit network with synthetic data elucidates the impact of hyperparameters on convergence rates and computational efficiency.