UniKnow: A Unified Framework for Reliable Language Model Behavior across Parametric and External Knowledge
This work addresses the problem of unreliable knowledge integration in language models for researchers and practitioners, though it is incremental as it builds on prior knowledge integration efforts.
The paper tackles the challenge of achieving reliable knowledge utilization in language models when combining parametric and external knowledge, by introducing UniKnow, a unified framework for controlled evaluation across diverse knowledge scenarios like conflict, distraction, and absence, revealing that existing methods struggle to generalize and exhibit biases.
Language models often benefit from external knowledge beyond parametric knowledge. While this combination enhances performance, achieving reliable knowledge utilization remains challenging, as it requires assessing the state of each knowledge source based on the presence of relevant information. Yet, prior work on knowledge integration often overlooks this challenge by assuming ideal conditions and provides limited coverage of knowledge scenarios. To address this gap, we introduce UniKnow, a Unified framework for reliable LM behavior across parametric and external Knowledge. UniKnow enables controlled evaluation across knowledge scenarios such as knowledge conflict, distraction, and absence conditions that are rarely addressed together. Beyond evaluating existing methods under this setting, we extend our work by introducing UniKnow-Aware methods to support comprehensive evaluation. Experiments on UniKnow reveal that existing methods struggle to generalize across a broader range of knowledge configurations and exhibit scenario-specific biases. UniKnow thus provides a foundation for systematically exploring and improving reliability under knowledge scenarios.