A Training-Time Diagnostic for Generalization via the Log-Alignment Ratio
This work provides a training-time diagnostic for generalization that requires no held-out validation data, addressing the problem of detecting overfitting in large-scale models.
The paper introduces the log-alignment ratio (LAR), a measure of parameter-activation alignment, and shows it tracks the transition from memorization to generalization in grokking and 3B-parameter language model pre-training. In grokking, LAR predicts the effective dimension of the learned function, and in language model pre-training, its deviation from a non-overfitting baseline tracks the generalization gap.
We study the log-alignment ratio (LAR), a measure of parameter-activation alignment, introduced in parameterization theory. We reformulate it as the overlap between a weight spectrum $p$ of the normalized squared singular values of a matrix and an activation spectrum $q$ of the normalized squared projections of inputs onto its singular directions. We show that unembedding LAR tracks the transition between memorization and generalization in two different settings by capturing the spread of $p$ and $q$ during training. In grokking, LAR predicts the effective dimension of the learned function: $k \approx n^{2(1-\text{LAR})}$, where $n$ is the input dimension of the matrix. In 3B-parameter language model pre-training, its deviation from a non-overfitting baseline tracks the generalization gap, and its rate of decline increases as overfitting approaches. LAR is computable from quantities available during the forward pass with negligible computational overhead, and requires no held-out validation data.