A Generalized Spectral Framework to Expain Neural Scaling and Compression Dynamics
This work provides a theoretical foundation for understanding scaling and compression dynamics in neural networks, but it appears incremental as it builds on existing spectral analyses.
The paper tackles the problem of unifying empirical scaling laws for neural networks and model compression by developing a generalized spectral framework, which recovers existing theories and yields an invariant relation between learning and compression.
Empirical scaling laws describe how test loss and other performance metrics depend on model size, dataset size, and compute. While such laws are consistent within specific regimes, apparently distinct scaling behaviors have been reported for related settings such as model compression. Motivated by recent progress in spectral analyses of neural representations, this paper develops a \emph{generalized spectral framework} that unifies learning dynamics and compression phenomena under a common functional ansatz. We generalize the spectral evolution function from the linear kernel form $g(λt)=λt$ to an asymptotically polynomial function $g(λ,t;β)$, characterized by an effective spectral--temporal elasticity $ρ(β)$. This framework recovers existing lazy and feature-learning theories as special cases and yields an invariant relation between learning and compression