Unified Scaling Laws for Compressed Representations
This work addresses the computational cost problem in AI for researchers and practitioners by providing a framework to compare and optimize compressed model formats, though it is incremental as it builds on existing scaling laws and compression techniques.
The paper tackled the problem of predicting model performance when training occurs over various compressed representations, such as sparse or quantized formats, by developing a unified scaling law framework. The result was the identification of a simple capacity metric that robustly predicts parameter efficiency across multiple compressed representations, with empirical validation showing improved accuracy potential and better training algorithms for sparse-quantized formats.
Scaling laws have shaped recent advances in machine learning by enabling predictable scaling of model performance based on model size, computation, and data volume. Concurrently, the rise in computational cost for AI has motivated model compression techniques, notably quantization and sparsification, which have emerged to mitigate the steep computational demands associated with large-scale training and inference. This paper investigates the interplay between scaling laws and compression formats, exploring whether a unified scaling framework can accurately predict model performance when training occurs over various compressed representations, such as sparse, scalar-quantized, sparse-quantized or even vector-quantized formats. Our key contributions include validating a general scaling law formulation and showing that it is applicable both individually but also composably across compression types. Based on this, our main finding is demonstrating both theoretically and empirically that there exists a simple "capacity" metric -- based on the representation's ability to fit random Gaussian data -- which can robustly predict parameter efficiency across multiple compressed representations. On the practical side, we extend our formulation to directly compare the accuracy potential of different compressed formats, and to derive better algorithms for training over sparse-quantized formats.