Do Language Models Know When They're Hallucinating References?
This addresses the reliability and misinformation issues in language models for users relying on accurate references, though it is incremental as it focuses on a specific type of hallucination.
The paper tackles the problem of language models generating hallucinated book and article references by proposing consistency checks, such as querying the model about authors, to identify these hallucinations without external resources. The result shows that models like GPT-4 often produce inconsistent author lists for hallucinated references but accurately recall real ones, indicating they 'know' when hallucinating.
State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. Such inaccurate outputs not only undermine the reliability of these models but also limit their use and raise serious concerns about misinformation and propaganda. In this work, we focus on hallucinated book and article references and present them as the "model organism" of language model hallucination research, due to their frequent and easy-to-discern nature. We posit that if a language model cites a particular reference in its output, then it should ideally possess sufficient information about its authors and content, among other relevant details. Using this basic insight, we illustrate that one can identify hallucinated references without ever consulting any external resources, by asking a set of direct or indirect queries to the language model about the references. These queries can be considered as "consistency checks." Our findings highlight that while LMs, including GPT-4, often produce inconsistent author lists for hallucinated references, they also often accurately recall the authors of real references. In this sense, the LM can be said to "know" when it is hallucinating references. Furthermore, these findings show how hallucinated references can be dissected to shed light on their nature. Replication code and results can be found at https://github.com/microsoft/hallucinated-references.