On the Persistent Effects of Lexicality in Large Language Mod
For researchers and practitioners using LLM representations, this work reveals a fundamental limitation where lexical cues dominate semantics, potentially undermining tasks relying on semantic understanding.
The paper quantifies the persistent influence of lexical overlap over semantic content in LLM representations across architectures and training regimes, finding a mid-depth region where both signals degrade. This lexical effect impacts downstream tasks like summarization and model editing.
Representations extracted from large language models (LLMs) play an important role in many downstream applications. However, the structure of these representations is often influenced by lexical overlap rather than semantic content. Our understanding of the relationship between this lexical influence and semantic content, and its implications for downstream tasks, remains limited. In this work, we investigate representations to quantify the effect of lexical overlap relative to semantic content. We consider several adversarial semantic stress tests and further connect our findings to the information theory perspective. We find that lexical influence extends across the depth of models, consistently across architectures, training regimes, and objective functions, including the models trained for semantic similarity. Moreover, we observe a mid-depth region in which both lexical and semantic signals degrade simultaneously, indicating a transitional regime where representations are poor for both surface form and meaning. We further demonstrate the effect of lexical influence on downstream uses of LLMs using summarization and model editing as a case study.