Neural Normalized Compression Distance and the Disconnect Between Compression and Classification
This work addresses a foundational disconnect in machine learning theory, with implications for understanding neural network compression and classification, though it is incremental in questioning established assumptions.
The paper investigates the relationship between compression and classification by developing a Neural Normalized Compression Distance using large language models as compressors, finding that classification accuracy does not correlate with compression rate, challenging existing intuitions.
It is generally well understood that predictive classification and compression are intrinsically related concepts in information theory. Indeed, many deep learning methods are explained as learning a kind of compression, and that better compression leads to better performance. We interrogate this hypothesis via the Normalized Compression Distance (NCD), which explicitly relies on compression as the means of measuring similarity between sequences and thus enables nearest-neighbor classification. By turning popular large language models (LLMs) into lossless compressors, we develop a Neural NCD and compare LLMs to classic general-purpose algorithms like gzip. In doing so, we find that classification accuracy is not predictable by compression rate alone, among other empirical aberrations not predicted by current understanding. Our results imply that our intuition on what it means for a neural network to ``compress'' and what is needed for effective classification are not yet well understood.