Structured second-order methods via natural gradient descent
This work provides incremental improvements to optimization methods for machine learning practitioners.
The authors tackled the problem of designing efficient optimization algorithms by proposing structured second-order and adaptive-gradient methods derived from natural-gradient descent on structured parameter spaces, achieving competitive performance on deterministic non-convex and deep learning problems.
In this paper, we propose new structured second-order methods and structured adaptive-gradient methods obtained by performing natural-gradient descent on structured parameter spaces. Natural-gradient descent is an attractive approach to design new algorithms in many settings such as gradient-free, adaptive-gradient, and second-order methods. Our structured methods not only enjoy a structural invariance but also admit a simple expression. Finally, we test the efficiency of our proposed methods on both deterministic non-convex problems and deep learning problems.