GenCRF: Generative Clustering and Reformulation Framework for Enhanced Intent-Driven Information Retrieval
This work addresses the challenge of enhancing search success rates for users in information retrieval, though it appears incremental as it builds on existing LLM-based query reformulation methods.
The paper tackles the problem of limited and redundant query expansions in intent-driven information retrieval by proposing GenCRF, a framework that generates and clusters queries to capture diverse intents, achieving state-of-the-art performance with up to 12% improvement in nDCG@10 on the BEIR benchmark.
Query reformulation is a well-known problem in Information Retrieval (IR) aimed at enhancing single search successful completion rate by automatically modifying user's input query. Recent methods leverage Large Language Models (LLMs) to improve query reformulation, but often generate limited and redundant expansions, potentially constraining their effectiveness in capturing diverse intents. In this paper, we propose GenCRF: a Generative Clustering and Reformulation Framework to capture diverse intentions adaptively based on multiple differentiated, well-generated queries in the retrieval phase for the first time. GenCRF leverages LLMs to generate variable queries from the initial query using customized prompts, then clusters them into groups to distinctly represent diverse intents. Furthermore, the framework explores to combine diverse intents query with innovative weighted aggregation strategies to optimize retrieval performance and crucially integrates a novel Query Evaluation Rewarding Model (QERM) to refine the process through feedback loops. Empirical experiments on the BEIR benchmark demonstrate that GenCRF achieves state-of-the-art performance, surpassing previous query reformulation SOTAs by up to 12% on nDCG@10. These techniques can be adapted to various LLMs, significantly boosting retriever performance and advancing the field of Information Retrieval.