Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Models
This addresses context integration issues in LLMs for applications like QA, but it is incremental as it builds on existing attention mechanisms.
The paper tackled the problem of context faithfulness hallucinations in large language models, where outputs deviate from retrieved information, by proposing Dynamic Attention-Guided Context Decoding (DAGCD), which improved faithfulness and robustness in open-book QA datasets while maintaining computational efficiency.
Large language models (LLMs) often exhibit Context Faithfulness Hallucinations, where outputs deviate from retrieved information due to incomplete context integration. Our analysis reveals a strong correlation between token-level uncertainty and hallucinations. We hypothesize that attention mechanisms inherently encode context utilization signals, supported by probing analysis. Based on these insights, we propose Dynamic Attention-Guided Context Decoding (DAGCD), a lightweight framework that leverages attention distributions and uncertainty signals in a single-pass decoding. Experiments on open-book QA datasets demonstrate DAGCD's effectiveness, yielding significant improvements in faithfulness and robustness while preserving computational efficiency.