A Theory of Generalization in Deep Learning
For deep learning practitioners and theorists, this work provides a unified theoretical framework for generalization and a practical algorithm that improves training efficiency and robustness.
The paper presents a non-asymptotic theory of generalization in deep learning based on the empirical neural tangent kernel, showing that signal and noise are separated into distinct channels. The theory explains phenomena like benign overfitting and double descent, and yields a practical SNR preconditioner that accelerates grokking by 5×, suppresses memorization, and improves DPO fine-tuning while staying 3× closer to the reference policy.
We present a non-asymptotic theory of generalization in deep learning where the empirical neural tangent kernel partitions the output space. In directions corresponding to signal, error dissipates rapidly; in the vast orthogonal dimensions corresponding to noise, the kernel's near-zero eigenvalues trap residual error in a test-invisible reservoir. Within the signal channel, minibatch SGD ensures that coherent population signal accumulates via fast linear drift, while idiosyncratic memorization is suppressed into a slow, diffusive random walk. We prove generalization survives even when the kernel evolves $\mathcal{O}(1)$ in operator norm, the full feature-learning regime. This theory naturally explains disparate phenomena in deep learning theory, such as benign overfitting, double descent, implicit bias, and grokking. Lastly, we derive an exact population-risk objective from a single training run with no validation data, for any architecture, loss, or optimizer, and prove that it measures precisely the noise in the signal channel. This objective reduces in practice to an SNR preconditioner on top of Adam, adding one state vector at no extra cost; it accelerates grokking by $5 \times$, suppresses memorization in PINNs and implicit neural representations, and improves DPO fine-tuning under noisy preferences while staying $3 \times$ closer to the reference policy.