Emergent Inference-Time Semantic Contamination via In-Context Priming
This work addresses security risks in LLM-based applications using few-shot prompting, identifying boundary conditions for inference-time contamination.
The study revisited the claim that few-shot prompting alone does not cause emergent misalignment in large language models, finding that inference-time semantic drift is measurable and requires models with sufficient capability, leading to significant distributional shifts toward harmful themes in larger models.
Recent work has shown that fine-tuning large language models (LLMs) on insecure code or culturally loaded numeric codes can induce emergent misalignment, causing models to produce harmful content in unrelated downstream tasks. The authors of that work concluded that $k$-shot prompting alone does not induce this effect. We revisit this conclusion and show that inference-time semantic drift is real and measurable; however, it requires models of large-enough capability. Using a controlled experiment in which five culturally loaded numbers are injected as few-shot demonstrations before a semantically unrelated prompt, we find that models with richer cultural-associative representations exhibit significant distributional shifts toward darker, authoritarian, and stigmatized themes, while a simpler/smaller model does not. We additionally find that structurally inert demonstrations (nonsense strings) perturb output distributions, suggesting two separable mechanisms: structural format contamination and semantic content contamination. Our results map the boundary conditions under which inference-time contamination occurs, and carry direct implications for the security of LLM-based applications that use few-shot prompting.