Breaking Token Into Concepts: Exploring Extreme Compression in Token Representation Via Compositional Shared Semantics
This addresses the need for more efficient and semantically rich token representations in NLP, offering a novel approach with broad applicability, though it builds on existing techniques like Product Quantization.
The paper tackled the problem of monolithic token embeddings in language models by proposing Aggregate Semantic Grouping (ASG), a compositional representation method, which achieved extreme compression of embedding parameters to 0.4-0.5% while maintaining over 95% task performance across diverse benchmarks.
Standard language models employ unique, monolithic embeddings for each token, potentially limiting their ability to capture the multifaceted nature of word meanings. We investigate whether tokens can be more effectively represented through a compositional structure that accumulates diverse semantic facets. To explore this, we propose Aggregate Semantic Grouping (ASG), a novel approach leveraging Product Quantization (PQ). We apply ASG to standard transformer architectures (mBERT, XLM-R, mT5) and evaluate this representational scheme across diverse tasks (NLI, NER, QA), as well as a biomedical domain-specific benchmark (BC5CDR) using BioBERT. Our findings demonstrate that representing tokens compositionally via ASG achieves extreme compression in embedding parameters (0.4--0.5\%) while maintaining $>$95\% task performance relative to the base model, even in generative tasks and extends to both cross lingual transfer and domain-specific settings. These results validate the principle that tokens can be effectively modeled as combinations of shared semantic building blocks. ASG offers a simple yet concrete method for achieving this, showcasing how compositional representations can capture linguistic richness while enabling compact yet semantically rich models.