Component-Wise Natural Gradient Descent -- An Efficient Neural Network Optimization
This addresses the problem of slow training in neural networks for researchers and practitioners by providing a more efficient optimization method, though it is incremental as it builds on existing NGD variants.
The paper tackles the computational inefficiency of Natural Gradient Descent (NGD) by proposing Component-Wise Natural Gradient Descent (CW-NGD), which decomposes the Fisher Information Matrix into smaller, invertible blocks for dense and convolutional layers, resulting in fewer iterations to converge compared to state-of-the-art methods.
Natural Gradient Descent (NGD) is a second-order neural network training that preconditions the gradient descent with the inverse of the Fisher Information Matrix (FIM). Although NGD provides an efficient preconditioner, it is not practicable due to the expensive computation required when inverting the FIM. This paper proposes a new NGD variant algorithm named Component-Wise Natural Gradient Descent (CW-NGD). CW-NGD is composed of 2 steps. Similar to several existing works, the first step is to consider the FIM matrix as a block-diagonal matrix whose diagonal blocks correspond to the FIM of each layer's weights. In the second step, unique to CW-NGD, we analyze the layer's structure and further decompose the layer's FIM into smaller segments whose derivatives are approximately independent. As a result, individual layers' FIMs are approximated in a block-diagonal form that trivially supports the inversion. The segment decomposition strategy is varied by layer structure. Specifically, we analyze the dense and convolutional layers and design their decomposition strategies appropriately. In an experiment of training a network containing these 2 types of layers, we empirically prove that CW-NGD requires fewer iterations to converge compared to the state-of-the-art first-order and second-order methods.