The SMeL Test: A simple benchmark for media literacy in language models
This work addresses the problem of media literacy in language models for AI researchers and developers, highlighting a critical form of hallucination, but it is incremental as it focuses on benchmarking rather than proposing new solutions.
The authors introduced the Synthetic Media Literacy Test (SMeL Test) to evaluate language models' ability to filter untrustworthy information, finding that no model consistently succeeded, with the best API model hallucinating up to 70% of the time and larger models not necessarily outperforming smaller ones.
The internet is rife with unattributed, deliberately misleading, or otherwise untrustworthy content. Though large language models (LLMs) are often tasked with autonomous web browsing, the extent to which they have learned the simple heuristics human researchers use to navigate this noisy environment is not currently known. In this paper, we introduce the Synthetic Media Literacy Test (SMeL Test), a minimal benchmark that tests the ability of language models to actively filter out untrustworthy information in context. We benchmark a variety of commonly used instruction-tuned LLMs, including reasoning models, and find that no model consistently succeeds; while reasoning in particular is associated with higher scores, even the best API model we test hallucinates up to 70% of the time. Remarkably, larger and more capable models do not necessarily outperform their smaller counterparts. We hope our work sheds more light on this important form of hallucination and guides the development of new methods to combat it.