Lower Layers Matter: Alleviating Hallucination via Multi-Layer Fusion Contrastive Decoding with Truthfulness Refocused
This addresses the issue of inaccurate outputs in LLMs for users relying on reliable text generation, though it is incremental as it builds on prior contrastive decoding methods.
The paper tackles the problem of hallucinations in Large Language Models by introducing a novel contrastive decoding framework called LOL, which integrates lower layers and a truthfulness refocused module, resulting in significant mitigation of hallucinations and outperforming existing baselines on four datasets.
Large Language Models (LLMs) have demonstrated exceptional performance across various natural language processing tasks. However, they occasionally generate inaccurate and counterfactual outputs, a phenomenon commonly referred to as "hallucinations''. To tackle this issue, recent studies have explored contrastive decoding between the original model and an amateur model with induced hallucination, showing promising results. Nevertheless, this approach can disrupt the original LLM's output distribution due to coarse contrast and simple subtraction operations, potentially leading to errors. In this paper, we introduce a novel contrastive decoding framework, termed LOL (LOwer Layer Matters). Unlike prior methods that focus solely on the final layer, our approach integrates contrastive information from lower layers to enable multi-layer fusion during contrastive decoding. Additionally, we incorporate a truthfulness refocused module that leverages instruction guidance to further improve truthfulness in contrastive decoding. Extensive experiments on four publicly available datasets demonstrate that the LOL framework significantly mitigates hallucination while outperforming existing baselines in most cases. For reproducibility, we will release our code and data upon acceptance.