An Effective Gram Matrix Characterizes Generalization in Deep Networks
This work provides a theoretical framework for understanding generalization in deep networks, which is incremental but offers new insights for researchers in machine learning.
The authors derived a differential equation to model the generalization gap evolution during gradient descent training, identifying an 'effective Gram matrix' that predicts test loss based on alignment with initial residuals. Empirical results on image classification show accurate test loss prediction and reveal that residuals align with small eigenvalue subspaces, indicating benign training.
We derive a differential equation that governs the evolution of the generalization gap when a deep network is trained by gradient descent. This differential equation is controlled by two quantities, a contraction factor that brings together trajectories corresponding to slightly different datasets, and a perturbation factor that accounts for them training on different datasets. We analyze this differential equation to compute an ``effective Gram matrix'' that characterizes the generalization gap in terms of the alignment between this Gram matrix and a certain initial ``residual''. Empirical evaluations on image classification datasets indicate that this analysis can predict the test loss accurately. Further, during training, the residual predominantly lies in the subspace of the effective Gram matrix with the smallest eigenvalues. This indicates that the generalization gap accumulates slowly along the direction of training, charactering a benign training process. We provide novel perspectives for explaining the generalization ability of neural network training with different datasets and architectures through the alignment pattern of the ``residual" and the ``effective Gram matrix".