CAAD: Context-Aware Adaptive Decoding for Truthful Text Generation
This addresses the problem of generating truthful text for users of LLMs, offering a scalable and efficient alternative to data-intensive methods, though it is incremental as it builds on existing decoding-time interventions.
The paper tackles the challenge of ensuring truthfulness in large language models by proposing a context-aware adaptive decoding method that uses a compact reference grounding space built from minimal annotated examples to shape logits during inference, achieving a 2.8% average improvement on TruthfulQA and outperforming baselines on other benchmarks.
Ensuring truthfulness in large language models remains a critical challenge for reliable text generation. While supervised fine-tuning and reinforcement learning with human feedback have shown promise, they require substantial amount of annotated data and computational resources, limiting scalability. In contrast, decoding-time interventions offer lightweight alternatives without model retraining. However, existing decoding strategies often face issues like prompt sensitivity, limited generalization, or dependence on internal model states. We propose a context-aware adaptive decoding method that leverages a compact reference grounding space, built from as few as 10 annotated examples and comprising pairs of context embeddings and next token logits from truthful responses, to enable retrieval-based logit shaping during inference. At each decoding step, our method retrieves top-N semantically similar contexts and aggregates their associated next token logits to modify the LLM's logits. Across three open-ended question-answering benchmarks, our approach achieves a 2.8 percent average improvement on TruthfulQA and further outperforms existing baselines on both Biographies and WikiQA. Experimental results also demonstrate cross-task generalization, with TruthfulQA-derived grounding enhancing biography generation. Our model-agnostic, scalable, and efficient method requires only a single generation pass, highlighting the potential of context-aware decoding for factual reliability in LLMs.