Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold
This addresses the problem of understanding delayed generalization in machine learning, particularly for researchers studying optimization dynamics, but it is incremental as it builds on prior geometric studies by incorporating noise and curvature interactions.
This paper tackles the problem of explaining grokking, where generalization occurs long after training convergence, by analyzing AdamW dynamics on modular arithmetic tasks, revealing a 'Spectral Gating' mechanism that regulates the transition from memorization to generalization. The result includes identifying three complexity regimes, such as the Variance-Limited Regime where generalization waits for the spectral gate to open, with specific thresholds like P ≈ 41.
Standard optimization theories struggle to explain grokking, where generalization occurs long after training convergence. While geometric studies attribute this to slow drift, they often overlook the interaction between the optimizer's noise structure and landscape curvature. This work analyzes AdamW dynamics on modular arithmetic tasks, revealing a ``Spectral Gating'' mechanism that regulates the transition from memorization to generalization. We find that AdamW operates as a variance-gated stochastic system. Grokking is constrained by a stability condition: the generalizing solution resides in a sharp basin ($λ_{max}^H$) initially inaccessible under low-variance regimes. The ``delayed'' phase represents the accumulation of gradient variance required to lift the effective stability ceiling, permitting entry into this sharp manifold. Our ablation studies identify three complexity regimes: (1) \textbf{Capacity Collapse} ($P < 23$), where rank-deficiency prevents structural learning; (2) \textbf{The Variance-Limited Regime} ($P \approx 41$), where generalization waits for the spectral gate to open; and (3) \textbf{Stability Override} ($P > 67$), where memorization becomes dimensionally unstable. Furthermore, we challenge the "Flat Minima" hypothesis for algorithmic tasks, showing that isotropic noise injection fails to induce grokking. Generalization requires the \textit{anisotropic rectification} unique to adaptive optimizers, which directs noise into the tangent space of the solution manifold.