Efficient OpAmp Adaptation for Zoom Attention to Golden Contexts
This addresses a specific issue in retrieval-augmented generation and long-context applications for AI researchers and practitioners, but it is incremental as it builds on existing Transformer architectures with adapters.
The paper tackles the problem of large language models being distracted by noisy reference documents in question-answering tasks, and the result is that their Qwen2.5-OpAmp-72B model surpasses state-of-the-art LLMs like DeepSeek-V3 and GPT-4o on noisy-context benchmarks.
Large language models (LLMs) have shown significant promise in question-answering (QA) tasks, particularly in retrieval-augmented generation (RAG) scenarios and long-context applications. However, their performance is hindered by noisy reference documents, which often distract from essential information. Despite fine-tuning efforts, Transformer-based architectures struggle to prioritize relevant content. This is evidenced by their tendency to allocate disproportionate attention to irrelevant or later-positioned documents. Recent work proposes the differential attention mechanism to address this issue, but this mechanism is limited by an unsuitable common-mode rejection ratio (CMRR) and high computational costs. Inspired by the operational amplifier (OpAmp), we propose the OpAmp adaptation to address these challenges, which is implemented with adapters efficiently. By integrating the adapter into pre-trained Transformer blocks, our approach enhances focus on the golden context without costly training from scratch. Empirical evaluations on noisy-context benchmarks reveal that our Qwen2.5-OpAmp-72B model, trained with our OpAmp adaptation, surpasses the performance of state-of-the-art LLMs, including DeepSeek-V3 and GPT-4o.