AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge
This addresses a key issue for LLM users in tasks like QA and summarization by improving handling of variable conflict scenarios, though it is incremental over existing contrastive methods.
The paper tackles the problem of knowledge conflict in large language models, where discrepancies between contextual and parametric knowledge hurt performance, by proposing AdaCAD, a fine-grained decoding method that dynamically adjusts based on conflict degree, achieving average QA accuracy gains of 14.21% over baselines and improving summary factuality by 6.19 AlignScore.
Knowledge conflict arises from discrepancies between information in the context of a large language model (LLM) and the knowledge stored in its parameters. This can hurt performance when using standard decoding techniques, which tend to ignore the context. Existing test-time contrastive methods seek to address this by comparing the LLM's output distribution with and without the context and adjust the model according to the contrast between them. However, we find that these methods frequently misjudge the degree of conflict and struggle to handle instances that vary in their amount of conflict, with static methods over-adjusting when conflict is absent. We propose a fine-grained, instance-level approach called AdaCAD, which dynamically infers the weight of adjustment based on the degree of conflict, as measured by the Jensen-Shannon divergence between distributions representing contextual and parametric knowledge. Across four LLMs, six question-answering (QA) and three summarization datasets, we demonstrate that ADACAD consistently outperforms other decoding baselines with average QA accuracy gains of 14.21% (absolute) over a static contrastive baseline, and improves the factuality of summaries by 6.19 (AlignScore). Lastly, we show that while contrastive baselines hurt performance when conflict is absent, ADACAD mitigates these losses, making it more applicable to real-world datasets in which some examples have conflict and others do not.