Can Synthetic Query Rewrites Capture User Intent Better than Humans in Retrieval-Augmented Generation?
This addresses the challenge of capturing user intent in RAG systems for improved retrieval and generation, offering a scalable alternative to human annotations, though it is incremental as it builds on existing query rewriting methods.
The paper tackles the problem of query rewriting in multi-turn RAG systems, where human rewrites often diverge from user intent, and shows that synthetic rewrites generated by SynRewrite outperform human rewrites, achieving superior performance in retrieval and generation tasks on datasets like TopiOCQA and QRECC.
Multi-turn RAG systems often face queries with colloquial omissions and ambiguous references, posing significant challenges for effective retrieval and generation. Traditional query rewriting relies on human annotators to clarify queries, but due to limitations in annotators' expressive ability and depth of understanding, manually rewritten queries often diverge from those needed in real-world RAG systems, resulting in a gap between user intent and system response. We observe that high-quality synthetic queries can better bridge this gap, achieving superior performance in both retrieval and generation compared to human rewrites. This raises an interesting question: Can rewriting models trained on synthetic queries better capture user intent than human annotators? In this paper, we propose SynRewrite, a synthetic data-driven query rewriting model to generate high-quality synthetic rewrites more aligned with user intent. To construct training data, we prompt GPT-4o with dialogue history, current queries, positive documents, and answers to synthesize high-quality rewrites. A Flan-T5 model is then finetuned on this dataset to map dialogue history and queries to synthetic rewrites. Finally, we further enhance the rewriter using the generator's feedback through the DPO algorithm to boost end-task performance. Experiments on TopiOCQA and QRECC datasets show that SynRewrite consistently outperforms human rewrites in both retrieval and generation tasks. Our results demonstrate that synthetic rewrites can serve as a scalable and effective alternative to human annotations.