Natural Gradient Methods: Perspectives, Efficient-Scalable Approximations, and Analysis
This work addresses the computational infeasibility of second-order methods like natural gradient descent in large-scale deep learning problems, offering insights into approximations for practical use.
The paper examines the natural gradient method as an alternative to stochastic gradient descent for faster convergence, reviewing its perspectives, efficient-scalable approximations, and analyzing their performance through experiments.
Natural Gradient Descent, a second-degree optimization method motivated by the information geometry, makes use of the Fisher Information Matrix instead of the Hessian which is typically used. However, in many cases, the Fisher Information Matrix is equivalent to the Generalized Gauss-Newton Method, that both approximate the Hessian. It is an appealing method to be used as an alternative to stochastic gradient descent, potentially leading to faster convergence. However, being a second-order method makes it infeasible to be used directly in problems with a huge number of parameters and data. This is evident from the community of deep learning sticking with the stochastic gradient descent method since the beginning. In this paper, we look at the different perspectives on the natural gradient method, study the current developments on its efficient-scalable empirical approximations, and finally examine their performance with extensive experiments.