Harnessing the Unseen: The Hidden Influence of Intrinsic Knowledge in Long-Context Language Models
This addresses a gap in evaluating long-context models for researchers and practitioners, though it is incremental as it builds on existing retrieval tests.
The paper investigates how intrinsic knowledge affects content generation in long-context language models, finding that its impact increases with context length and that better extrinsic retrieval can interfere with intrinsic knowledge use. It introduces a Hybrid Needle-in-a-Haystack test, showing Qwen-2.5 models outperform Llama-3.1 models in intrinsic retrieval ability, with Llama-3.1-70B-Instruct failing to improve under long-context conditions.
Recent advances in long-context models (LCMs), designed to handle extremely long input contexts, primarily focus on utilizing external contextual information, often leaving the influence of large language models' intrinsic knowledge underexplored. In this work, we investigate how this intrinsic knowledge affects content generation and demonstrate that its impact becomes increasingly pronounced as context length extends. Furthermore, we show that the model's ability to utilize intrinsic knowledge, which we call intrinsic retrieval ability, does not improve simultaneously with its ability to leverage contextual knowledge through extrinsic retrieval ability. Moreover, better extrinsic retrieval can interfere with the model's ability to use its own knowledge effectively, limiting its full potential. To bridge this gap, we design a simple yet effective Hybrid Needle-in-a-Haystack test that evaluates models based on their capabilities across both retrieval abilities, rather than solely emphasizing extrinsic retrieval ability. Our experimental results reveal that Qwen-2.5 models significantly outperform Llama-3.1 models, demonstrating superior intrinsic retrieval ability. Moreover, even the more powerful Llama-3.1-70B-Instruct model fails to exhibit better performance under LCM conditions, highlighting the importance of evaluating models from a dual-retrieval perspective.