Compression Hacking: A Supplementary Perspective on Informatics Properties of Language Models from Geometric Distortion
This work addresses the issue of misleading compression metrics in language models for researchers and practitioners, offering an incremental improvement by refining existing approaches.
The paper tackles the problem that highly compressed language models (LMs) suffer from geometric anisotropy, which degrades their performance, by proposing refined compression metrics that incorporate geometric distortion analysis. The result is that these metrics achieve Spearman correlation coefficients above 0.9 with LM capabilities, significantly outperforming existing metrics.
Recently, the concept of ``compression as intelligence'' has provided a novel informatics metric perspective for language models (LMs), emphasizing that highly structured representations signify the intelligence level of LMs. However, from a geometric standpoint, the word representation space of highly compressed LMs tends to degenerate into a highly anisotropic state, which hinders the LM's ability to comprehend instructions and directly impacts its performance. We found this compression-anisotropy synchronicity is essentially the ``Compression Hacking'' in LM representations, where noise-dominated directions tend to create the illusion of high compression rates by sacrificing spatial uniformity. Based on this, we propose three refined compression metrics by incorporating geometric distortion analysis and integrate them into a self-evaluation pipeline. The refined metrics exhibit strong alignment with the LM's comprehensive capabilities, achieving Spearman correlation coefficients above 0.9, significantly outperforming both the original compression and other internal structure-based metrics. This confirms that compression hacking substantially enhances the informatics interpretation of LMs by incorporating geometric distortion of representations.