HalluHard: A Hard Multi-Turn Hallucination Benchmark
This addresses the issue of unreliable factual claims in AI systems for users in critical domains like law, medicine, and research, though it is incremental as it builds on existing hallucination benchmarks.
The authors tackled the problem of factual hallucinations in large language models during multi-turn dialogues by introducing HalluHard, a challenging benchmark with 950 questions across high-stakes domains, and found that even top models with web search still hallucinate around 30% of the time.
Large language models (LLMs) still produce plausible-sounding but ungrounded factual claims, a problem that worsens in multi-turn dialogue as context grows and early errors cascade. We introduce $\textbf{HalluHard}$, a challenging multi-turn hallucination benchmark with 950 seed questions spanning four high-stakes domains: legal cases, research questions, medical guidelines, and coding. We operationalize groundedness by requiring inline citations for factual assertions. To support reliable evaluation in open-ended settings, we propose a judging pipeline that iteratively retrieves evidence via web search. It can fetch, filter, and parse full-text sources (including PDFs) to assess whether cited material actually supports the generated content. Across a diverse set of frontier proprietary and open-weight models, hallucinations remain substantial even with web search ($\approx 30\%$ for the strongest configuration, Opus-4.5 with web search), with content-grounding errors persisting at high rates. Finally, we show that hallucination behavior is shaped by model capacity, turn position, effective reasoning, and the type of knowledge required.