ReFilter: Improving Robustness of Retrieval-Augmented Generation via Gated Filter
This work improves robustness and efficiency in retrieval-augmented generation for knowledge-intensive question answering, particularly in domains like biomedicine, though it is incremental as it builds on existing fusion methods.
The paper tackles the problem of scaling retrieval-augmented generation (RAG) for question answering by addressing inefficiencies from irrelevant retrieved content as the number of candidates increases, proposing ReFilter, a latent-based fusion framework that achieves state-of-the-art performance with 70.01% average accuracy on biomedical benchmarks in zero-shot transfer.
Retrieval-augmented generation (RAG) has become a dominant paradigm for grounding large language models (LLMs) with external evidence in knowledge-intensive question answering. A core design choice is how to fuse retrieved samples into the LLMs, where existing internal fusion approaches broadly fall into query-based fusion, parametric fusion, and latent-based fusion. Despite their effectiveness at modest retrieval scales, these methods often fail to scale gracefully as the number of retrieved candidates k increases: Larger k improves evidence coverage, yet realistic top-k retrieval inevitably contains irrelevant or redundant content and increases the inference cost. To address these limitations, we propose ReFilter, a novel latent-based fusion framework that performs token-level filtering and fusion. ReFilter consists of three key components: a context encoder for encoding context features, a gated filter for weighting each token, and a token fusion module for integrating the weighted token feature into the LLM's hidden states. Our experiments across four general-domain QA benchmarks show that ReFilter consistently achieves the best average performance under both in-domain adaptation and out-of-domain transfer. ReFilter further generalizes to five biomedical QA benchmarks in zero-shot transfer without domain fine-tuning, reaching 70.01% average accuracy with Qwen2.5-14B-Instruct.