Learning the Wrong Lessons: Syntactic-Domain Spurious Correlations in Language Models
This addresses a subtle but impactful issue for AI safety and reliability, as it reveals how models can learn unintended biases from training data, potentially compromising their robustness and security.
The paper tackles the problem of spurious correlations between syntax and domain in language models, which can cause models to learn incorrect associations that override prompt semantics and lower performance on tasks like entity knowledge by a mean of 0.51 +/- 0.06. It also shows these correlations can be exploited to bypass safety refusals in models like OLMo-2-7B Instruct and GPT-4o.
For an LLM to correctly respond to an instruction it must understand both the semantics and the domain (i.e., subject area) of a given task-instruction pair. However, syntax can also convey implicit information Recent work shows that syntactic templates -- frequent sequences of Part-of-Speech (PoS) tags -- are prevalent in training data and often appear in model outputs. In this work we characterize syntactic templates, domain, and semantics in task-instruction pairs. We identify cases of spurious correlations between syntax and domain, where models learn to associate a domain with syntax during training; this can sometimes override prompt semantics. Using a synthetic training dataset, we find that the syntactic-domain correlation can lower performance (mean 0.51 +/- 0.06) on entity knowledge tasks in OLMo-2 models (1B-13B). We introduce an evaluation framework to detect this phenomenon in trained models, and show that it occurs on a subset of the FlanV2 dataset in open (OLMo-2-7B; Llama-4-Maverick), and closed (GPT-4o) models. Finally, we present a case study on the implications for safety finetuning, showing that unintended syntactic-domain correlations can be used to bypass refusals in OLMo-2-7B Instruct and GPT-4o. Our findings highlight two needs: (1) to explicitly test for syntactic-domain correlations, and (2) to ensure syntactic diversity in training data, specifically within domains, to prevent such spurious correlations.