Sentence-Level Contextual Entrainment in Large Language Models
For LLM researchers and practitioners, this identifies a subtle bias mechanism and offers a targeted mitigation strategy, though the phenomenon is newly discovered and the work is incremental.
The paper extends contextual entrainment from token to sentence level in LLMs, showing that sentences in prompts (even counterfactual) increase their probability during inference. It finds that 2-4% of attention heads control this effect, and disabling them mitigates entrainment without harming performance.
Contextual entrainment, which is a newly discovered phenomenon in large language models (LLMs), refers to the tendency of a model to assign higher probabilities to tokens that appear in its context. In this work, we extend this phenomenon from the token level to the sentence level by examining the per-token mean log-probability of a sentence instead of the probabilities of individual tokens. We investigate sentence-level contextual entrainment across 26 LLMs from seven families and two datasets, which cover both subjective and objective tasks. We find that sentence-level contextual entrainment exists. This means that the sentences in the prompt (even if they are counterfactual statements) can significantly increase their probability during model inference time. As the model size increases, contextual entrainment gradually decreases. We also find that contextual entrainment is controlled by 2% to 4% of the attention heads. Turning off these attention heads can effectively mitigate contextual entrainment without hurting the model's performance.