A Common Framework for Natural Gradient and Taylor based Optimisation using Manifold Theory
This work offers a theoretical unification for optimization techniques in machine learning, but it appears incremental as it builds on existing methods without introducing new algorithms.
The authors developed a theoretical framework that connects Taylor approximation-based optimization methods with Natural Gradient (NG) optimization, which is Fisher efficient for probabilistic models, and provided mathematical justification for combining higher-order methods with NG.
This technical report constructs a theoretical framework to relate standard Taylor approximation based optimisation methods with Natural Gradient (NG), a method which is Fisher efficient with probabilistic models. Such a framework will be shown to also provide mathematical justification to combine higher order methods with the method of NG.