Is All Learning (Natural) Gradient Descent?
This provides a unifying theoretical framework for understanding diverse learning algorithms in machine learning, though it is incremental in nature.
The paper demonstrates that a broad class of effective learning rules, which improve a scalar performance measure over time, can be reformulated as natural gradient descent with respect to a defined loss function and metric, including identifying optimal metrics such as one with the minimum condition number.
This paper shows that a wide class of effective learning rules -- those that improve a scalar performance measure over a given time window -- can be rewritten as natural gradient descent with respect to a suitably defined loss function and metric. Specifically, we show that parameter updates within this class of learning rules can be expressed as the product of a symmetric positive definite matrix (i.e., a metric) and the negative gradient of a loss function. We also demonstrate that these metrics have a canonical form and identify several optimal ones, including the metric that achieves the minimum possible condition number. The proofs of the main results are straightforward, relying only on elementary linear algebra and calculus, and are applicable to continuous-time, discrete-time, stochastic, and higher-order learning rules, as well as loss functions that explicitly depend on time.