Learning as Observable Matrix Dynamics: Diffusive Relaxations versus Phase Transitions
For researchers studying neural network training dynamics, OMD provides a diagnostic tool that reveals geometric structure missed by scalar loss functions, though it is an incremental extension of existing random matrix theory and particle dynamics methods.
The paper introduces Observable Matrix Dynamics (OMD), a framework for diagnosing neural network training dynamics by analyzing spectral reorganizations of distance matrices on a fixed set of inputs. Across seven experiments, OMD distinguishes diffusive regimes (lacking stable band structure) from phase transitions (producing stable fingerprints), enabling identification of the geometric regime of representations.
Observable Matrix Dynamics (OMD) is a diagnostic framework that probes the dynamics of high-dimensional internal representations of inputs by a neural network via a fixed-size $N \times N$ distance matrix $M(t)$ on a held set of $N$ inputs. OMD uses methods of random matrix theory and particle dynamics to explore spectral reorganisations that are missed by scalar loss functions, but are informative of the training process. We read $M(t)$ against a perturbative ambient-versus-latent decomposition extending the Bogomolny--Bohigas--Schmit (BBS) theory of random distance matrices, with per-snapshot diagnostics for the top-of-spectrum band structure and ambient noise, trajectory-level observables linking snapshots, and a 3D MDS embedding (bottom-three eigenvectors) rendering training as a moving particle cloud. Across seven experiments, diffusive regimes lack stable top-of-spectrum band structure, while sharp endogenous or externally driven reorganisations produce stable fingerprints: consistent with smooth or product latent geometries in BBS-adjacent cases, and with finite-cluster or Fourier-soliton structures otherwise. OMD thus reads the geometric regime of a representation rather than reporting a single intrinsic dimension.