Failure of contextual invariance in gender inference with large language models
This reveals a failure of contextual invariance in LLMs, impacting bias benchmarking and deployment in high-stakes settings.
The study tested whether large language models (LLMs) maintain stable outputs under contextually equivalent formulations in a gender inference task, finding that minimal discourse context induced large, systematic shifts in outputs, with 19–52% of cases showing persistent dependence on context across models.
Standard evaluation practices assume that large language model (LLM) outputs are stable under contextually equivalent formulations of a task. Here, we test this assumption in the setting of gender inference. Using a controlled pronoun selection task, we introduce minimal, theoretically uninformative discourse context and find that this induces large, systematic shifts in model outputs. Correlations with cultural gender stereotypes, present in decontextualized settings, weaken or disappear once context is introduced, while theoretically irrelevant features, such as the gender of a pronoun for an unrelated referent, become the most informative predictors of model behaviour. A Contextuality-by-Default analysis reveals that, in 19--52\% of cases across models, this dependence persists after accounting for all marginal effects of context on individual outputs and cannot be attributed to simple pronoun repetition. These findings show that LLM outputs violate contextual invariance even under near-identical syntactic formulations, with implications for bias benchmarking and deployment in high-stakes settings.