A Mechanistic Understanding of Pronoun Fidelity in LLMs
For NLP researchers studying fairness and robustness in LLMs, this work provides a mechanistic understanding of pronoun fidelity, revealing that failures arise from competing causal subspaces rather than a single bias.
The paper investigates the internal mechanisms behind pronoun fidelity in LLMs, finding that three causal subspaces—group entity binding, recency bias, and stereotypical bias—coexist and together account for 91-99.5% of model behavior, with no single mechanism dominating.
Faithful and robust pronoun use is important for fair and coherent generations, yet large language models largely fail when multiple referents use different pronouns. To study the interplay of reasoning, repetition, and bias in this task, prior work relies exclusively on behavioural approaches, which may not reflect a model's internal workings. Therefore, we provide a mechanistic, model-internal perspective on pronoun fidelity, testing whether three mechanisms -- group entity binding (G), recency bias (R), and stereotypical bias (S) -- are causally implemented across several SOTA language models. Using Boundless Distributed Alignment Search, we find all three coexist as causal subspaces distributed across network depth. No single mechanism fully explains model behaviour, but a combination of the three consistently accounts for 91-99.5%. An attention head analysis further reveals two competing copying routes; group binding and stereotype share a localized concept-level route that retrieves a bound occupation-pronoun unit, while recency uses a distributed token-level route that repeats surface forms. In sum, pronoun fidelity arises from competition between simultaneously active causal subspaces.