Rethinking Exponential Averaging of the Fisher
This work addresses a common but poorly justified practice in machine learning optimization, offering a new theoretical perspective and algorithm, but it is incremental as it builds on existing methods with domain-specific improvements.
The paper tackles the lack of principled justification for exponential averaging in curvature-matrix estimates for optimization by connecting it to 'Wake of Quadratic regularized models' and proposing a new family of algorithms called KL-Divergence Wake-Regularized Models (KLD-WRM). The result shows that KLD-WRM outperforms K-FAC on MNIST, though no specific numerical gains are provided.
In optimization for Machine learning (ML), it is typical that curvature-matrix (CM) estimates rely on an exponential average (EA) of local estimates (giving EA-CM algorithms). This approach has little principled justification, but is very often used in practice. In this paper, we draw a connection between EA-CM algorithms and what we call a "Wake of Quadratic regularized models". The outlined connection allows us to understand what EA-CM algorithms are doing from an optimization perspective. Generalizing from the established connection, we propose a new family of algorithms, "KL-Divergence Wake-Regularized Models" (KLD-WRM). We give three different practical instantiations of KLD-WRM, and show numerically that these outperform K-FAC on MNIST.