WildFrame: Comparing Framing in Humans and LLMs on Naturally Occurring Texts
This work addresses the problem of understanding and comparing framing effects in LLMs and humans for model developers, enabling them to harness or mitigate these effects in applications, but it is incremental as it builds on prior work with synthetic data.
The study introduced WildFrame, a dataset of 1,000 naturally-occurring texts with positive and negative reframing, to evaluate framing effects in LLMs compared to humans, finding that all eight state-of-the-art LLMs exhibited framing effects similar to humans (r ≥ 0.57) and were more influenced by positive reframing.
Humans are influenced by how information is presented, a phenomenon known as the framing effect. Previous work has shown that LLMs may also be susceptible to framing but has done so on synthetic data and did not compare to human behavior. We introduce WildFrame, a dataset for evaluating LLM responses to positive and negative framing, in naturally-occurring sentences, and compare humans on the same data. WildFrame consists of 1,000 texts, first selecting real-world statements with clear sentiment, then reframing them in either positive or negative light, and lastly, collecting human sentiment annotations. By evaluating eight state-of-the-art LLMs on WildFrame, we find that all models exhibit framing effects similar to humans ($r\geq0.57$), with both humans and models being more influenced by positive rather than negative reframing. Our findings benefit model developers, who can either harness framing or mitigate its effects, depending on the downstream application.