Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models
This reveals a previously unknown bias in language models that affects generation patterns, which is incremental as it builds on existing bias research.
The authors identified a new phenomenon called semantic leakage in language models, where irrelevant prompt information leaks into generations, and measured significant leakage in 13 flagship models across languages and settings.
Despite their wide adoption, the biases and unintended behaviors of language models remain poorly understood. In this paper, we identify and characterize a phenomenon never discussed before, which we call semantic leakage, where models leak irrelevant information from the prompt into the generation in unexpected ways. We propose an evaluation setting to detect semantic leakage both by humans and automatically, curate a diverse test suite for diagnosing this behavior, and measure significant semantic leakage in 13 flagship models. We also show that models exhibit semantic leakage in languages besides English and across different settings and generation scenarios. This discovery highlights yet another type of bias in language models that affects their generation patterns and behavior.