Stochastic weight matrix dynamics during learning and Dyson Brownian motion
This provides a theoretical framework for understanding stochastic dynamics in machine learning, which is incremental but offers new insights for researchers in optimization and random matrix theory.
The paper demonstrates that weight matrix updates in learning algorithms can be described using Dyson Brownian motion, linking stochasticity to the learning rate and mini-batch size ratio, and identifies universal features like the Wigner surmise in specific models.
We demonstrate that the update of weight matrices in learning algorithms can be described in the framework of Dyson Brownian motion, thereby inheriting many features of random matrix theory. We relate the level of stochasticity to the ratio of the learning rate and the mini-batch size, providing more robust evidence to a previously conjectured scaling relationship. We discuss universal and non-universal features in the resulting Coulomb gas distribution and identify the Wigner surmise and Wigner semicircle explicitly in a teacher-student model and in the (near-)solvable case of the Gaussian restricted Boltzmann machine.