PFME: A Modular Approach for Fine-grained Hallucination Detection and Editing of Large Language Models
This addresses the issue of inaccurate content generation in LLMs for users relying on factual accuracy, though it appears incremental as it builds on existing detection and editing methods.
The paper tackles the problem of fine-grained hallucinations in Large Language Models by proposing PFME, a modular framework for detection and editing, which improves detection performance by 8.7 percentage points over ChatGPT and enhances editing scores by up to 16.2 percentage points on benchmark datasets.
Large Language Models (LLMs) excel in fluency but risk producing inaccurate content, called "hallucinations." This paper outlines a standardized process for categorizing fine-grained hallucination types and proposes an innovative framework--the Progressive Fine-grained Model Editor (PFME)--specifically designed to detect and correct fine-grained hallucinations in LLMs. PFME consists of two collaborative modules: the Real-time Fact Retrieval Module and the Fine-grained Hallucination Detection and Editing Module. The former identifies key entities in the document and retrieves the latest factual evidence from credible sources. The latter further segments the document into sentence-level text and, based on relevant evidence and previously edited context, identifies, locates, and edits each sentence's hallucination type. Experimental results on FavaBench and FActScore demonstrate that PFME outperforms existing methods in fine-grained hallucination detection tasks. Particularly, when using the Llama3-8B-Instruct model, PFME's performance in fine-grained hallucination detection with external knowledge assistance improves by 8.7 percentage points (pp) compared to ChatGPT. In editing tasks, PFME further enhances the FActScore of FActScore-Alpaca13B and FActScore-ChatGPT datasets, increasing by 16.2pp and 4.6pp, respectively.