Learnability Window in Gated Recurrent Neural Networks
This work addresses the fundamental challenge of understanding and improving long-term learning in recurrent neural networks for machine learning researchers, though it appears incremental as it builds on classical analyses of stability.
The paper tackles the problem of temporal learnability in recurrent neural networks by developing a statistical theory that shows how gating mechanisms determine the maximal temporal horizon for gradient information recovery, revealing scaling laws for logarithmic, polynomial, and exponential learning regimes based on gate-induced time-scale spectra and heavy-tailed noise.
We develop a statistical theory of temporal learnability in recurrent neural networks, showing how gating mechanisms determine the learnability window $\mathcal{H}_N$, defined as the maximal temporal horizon over which gradient information remains recoverable at sample size $N$. While classical analyses emphasize numerical stability of Jacobian products, we show that stability alone does not guarantee recoverability. Instead, learnability is governed by the interaction between the decay geometry of the effective learning rate envelope $f(\ell)=\|μ_{t,\ell}\|_1$, derived from first-order expansions of gate-induced Jacobians in Backpropagation Through Time, and the statistical concentration properties of stochastic gradients. Under heavy-tailed ($α$-stable) gradient noise, empirical averages concentrate at rate $N^{-1/κ_α}$ with $κ_α=α/(α-1)$. We prove that this interaction yields explicit scaling laws for the growth of $\mathcal{H}_N$, distinguishing logarithmic, polynomial, and exponential temporal learning regimes according to the attenuation of $f(\ell)$. The theory reveals that gate-induced time-scale spectra are the dominant determinants of temporal learnability: broader spectra slow envelope decay and systematically expand $\mathcal{H}_N$, whereas heavy-tailed noise uniformly compresses temporal horizons by weakening statistical concentration. Empirical results across multiple gated architectures confirm these structural scaling predictions.