Sherlock Your Queries: Learning to Ask the Right Questions for Dialogue-Based Retrieval
This work addresses inefficiencies in dialogue-based retrieval systems for users needing precise information, though it is incremental as it builds on existing interactive retrieval methods.
The paper tackles the problem of ambiguous user queries in information retrieval by proposing SherlockLLM, a dialogue-driven framework that learns an optimal questioning strategy using Reinforcement Learning, resulting in performance matching strong baselines on structured tasks and significantly outperforming them on unstructured tasks.
User queries in information retrieval are often ambiguous, making it challenging for systems to identify a user's target from a single query. While recent dialogue-based interactive retrieval systems can clarify user intent, they are inefficient as they often lack an explicit strategy to ask the most informative questions. To address this limitation, we propose SherlockLLM, a dialogue-driven retrieval framework that learns an optimal questioning strategy via Reinforcement Learning (RL) and avoids the need for large-scale annotated dialogue data. In our framework, an agent is trained to generate a sequence of binary questions to efficiently narrow down the search space. To validate our approach, we introduce a benchmark with both structured and unstructured tasks. Experimental results show that SherlockLLM is a robust and efficient solution. On the structured tasks, its performance matches strong baselines and approaches the theoretical optimal defined by binary search. On the challenging unstructured task, our agent significantly outperforms these baselines, showcasing its ability to learn a highly effective information-seeking dialogue policy.