Trustworthy AI for Medicine: Continuous Hallucination Detection and Elimination with CHECK
This addresses the critical issue of hallucinations for safe LLM deployment in high-stakes medical applications, representing a strong specific gain rather than an incremental improvement.
The authors tackled the problem of hallucinations in large language models for healthcare by developing CHECK, a continuous-learning framework that integrates clinical databases with an information-theoretic classifier, reducing hallucination rates from 31% to 0.3% on clinical trial questions and achieving a state-of-the-art 92.1% USMLE passing rate.
Large language models (LLMs) show promise in healthcare, but hallucinations remain a major barrier to clinical use. We present CHECK, a continuous-learning framework that integrates structured clinical databases with a classifier grounded in information theory to detect both factual and reasoning-based hallucinations. Evaluated on 1500 questions from 100 pivotal clinical trials, CHECK reduced LLama3.3-70B-Instruct hallucination rates from 31% to 0.3% - making an open source model state of the art. Its classifier generalized across medical benchmarks, achieving AUCs of 0.95-0.96, including on the MedQA (USMLE) benchmark and HealthBench realistic multi-turn medical questioning. By leveraging hallucination probabilities to guide GPT-4o's refinement and judiciously escalate compute, CHECK boosted its USMLE passing rate by 5 percentage points, achieving a state-of-the-art 92.1%. By suppressing hallucinations below accepted clinical error thresholds, CHECK offers a scalable foundation for safe LLM deployment in medicine and other high-stakes domains.