Generalized Reinforcement Learning for Retriever-Specific Query Rewriter with Unstructured Real-World Documents
This work addresses the problem of improving retrieval performance for RAG systems across varied domains, offering a scalable, annotation-free solution, though it is incremental with limitations in semantic and hybrid retrieval contexts.
The paper tackled the challenge of optimizing query formulation for Retrieval-Augmented Generation (RAG) systems with unstructured real-world documents by introducing RL-QR, a reinforcement learning framework for retriever-specific query rewriting, which achieved an 11% relative gain in NDCG@3 for multi-modal RAG and a 9% gain for lexical retrievers in experiments.
Retrieval-Augmented Generation (RAG) systems rely heavily on effective query formulation to unlock external knowledge, yet optimizing queries for diverse, unstructured real-world documents remains a challenge. We introduce \textbf{RL-QR}, a reinforcement learning framework for retriever-specific query rewriting that eliminates the need for human-annotated datasets and extends applicability to both text-only and multi-modal databases. By synthesizing scenario-question pairs and leveraging Generalized Reward Policy Optimization (GRPO), RL-QR trains query rewriters tailored to specific retrievers, enhancing retrieval performance across varied domains. Experiments on industrial in-house data demonstrate significant improvements, with $\text{RL-QR}_{\text{multi-modal}}$ achieving an 11\% relative gain in NDCG@3 for multi-modal RAG and $\text{RL-QR}_{\text{lexical}}$ yielding a 9\% gain for lexical retrievers. However, challenges persist with semantic and hybrid retrievers, where rewriters failed to improve performance, likely due to training misalignments. Our findings highlight RL-QR's potential to revolutionize query optimization for RAG systems, offering a scalable, annotation-free solution for real-world retrieval tasks, while identifying avenues for further refinement in semantic retrieval contexts.