Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory
This work addresses the problem of rigorously assessing compression limits for neural networks, which is incremental as it builds on existing theories to provide new insights.
The paper tackled the problem of evaluating neural network compressibility by extending the minimum description length principle to singular models using singular learning theory, finding that complexity estimates based on the local learning coefficient are closely correlated with compressibility in experiments on the Pythia suite.
We study neural network compressibility by using singular learning theory to extend the minimum description length (MDL) principle to singular models like neural networks. Through extensive experiments on the Pythia suite with quantization, factorization, and other compression techniques, we find that complexity estimates based on the local learning coefficient (LLC) are closely, and in some cases, linearly correlated with compressibility. Our results provide a path toward rigorously evaluating the limits of model compression.