Efficient Numerical Algorithm for Large-Scale Damped Natural Gradient Descent
This work addresses a computational bottleneck in natural gradient descent and stochastic reconfiguration for large-scale machine learning, though it appears incremental as it builds on existing methods.
The paper tackles the problem of efficiently solving the damped Fisher matrix in large-scale scenarios with many parameters and few samples, proposing a new algorithm based on Cholesky decomposition that is significantly faster than existing methods.
We propose a new algorithm for efficiently solving the damped Fisher matrix in large-scale scenarios where the number of parameters significantly exceeds the number of available samples. This problem is fundamental for natural gradient descent and stochastic reconfiguration. Our algorithm is based on Cholesky decomposition and is generally applicable. Benchmark results show that the algorithm is significantly faster than existing methods.