Breaking the Trade-Off Between Faithfulness and Expressiveness for Large Language Models
This addresses a key limitation in LLMs for applications requiring reliable and natural text generation, though it appears incremental as it builds on existing grounding strategies.
The paper tackles the problem of large language models struggling to integrate external knowledge while maintaining both faithfulness and expressiveness, proposing Collaborative Decoding (CoDe) to break this trade-off, with experiments showing superior performance in enhancing faithfulness without compromising expressiveness across diverse models and metrics.
Grounding responses in external knowledge represents an effective strategy for mitigating hallucinations in Large Language Models (LLMs). However, current LLMs struggle to seamlessly integrate knowledge while simultaneously maintaining faithfulness (or fidelity) and expressiveness, capabilities that humans naturally possess. This limitation results in outputs that either lack support from external knowledge, thereby compromising faithfulness, or appear overly verbose and unnatural, thus sacrificing expressiveness. In this work, to break the trade-off between faithfulness and expressiveness, we propose Collaborative Decoding (CoDe), a novel approach that dynamically integrates output probabilities generated with and without external knowledge. This integration is guided by distribution divergence and model confidence, enabling the selective activation of relevant and reliable expressions from the model's internal parameters. Furthermore, we introduce a knowledge-aware reranking mechanism that prevents over-reliance on prior parametric knowledge while ensuring proper utilization of provided external information. Through comprehensive experiments, our plug-and-play CoDe framework demonstrates superior performance in enhancing faithfulness without compromising expressiveness across diverse LLMs and evaluation metrics, validating both its effectiveness and generalizability.