Retrieval Meets Reasoning: Dynamic In-Context Editing for Long-Text Understanding
This addresses a key bottleneck for LLMs in handling extensive textual contexts, offering a pragmatic solution to enhance reasoning without high training costs, though it is incremental as it builds on retrieval-augmented generation and knowledge editing techniques.
The paper tackles the problem of large language models (LLMs) struggling with multi-hop reasoning in long texts due to fixed context lengths, and introduces a dynamic in-context editing method that improves performance, outperforming state-of-the-art context window extrapolation methods and competing with advanced commercial models.
Current Large Language Models (LLMs) face inherent limitations due to their pre-defined context lengths, which impede their capacity for multi-hop reasoning within extensive textual contexts. While existing techniques like Retrieval-Augmented Generation (RAG) have attempted to bridge this gap by sourcing external information, they fall short when direct answers are not readily available. We introduce a novel approach that re-imagines information retrieval through dynamic in-context editing, inspired by recent breakthroughs in knowledge editing. By treating lengthy contexts as malleable external knowledge, our method interactively gathers and integrates relevant information, thereby enabling LLMs to perform sophisticated reasoning steps. Experimental results demonstrate that our method effectively empowers context-limited LLMs, such as Llama2, to engage in multi-hop reasoning with improved performance, which outperforms state-of-the-art context window extrapolation methods and even compares favorably to more advanced commercial long-context models. Our interactive method not only enhances reasoning capabilities but also mitigates the associated training and computational costs, making it a pragmatic solution for enhancing LLMs' reasoning within expansive contexts.