Teaching Language Models To Gather Information Proactively
This addresses the problem of LLMs failing as collaborative partners in real-world settings due to passive responses, representing a novel task paradigm rather than an incremental improvement.
The paper tackles the problem of LLMs being passive when faced with incomplete prompts by introducing a proactive information gathering paradigm, where models identify context gaps and ask targeted questions to elicit implicit user knowledge. The result is a Qwen-2.5-7B model that outperforms o3-mini by 18% on automatic metrics and is favored by humans by 42% for clarification questions and 28% for final outlines.
Large language models (LLMs) are increasingly expected to function as collaborative partners, engaging in back-and-forth dialogue to solve complex, ambiguous problems. However, current LLMs often falter in real-world settings, defaulting to passive responses or narrow clarifications when faced with incomplete or under-specified prompts, falling short of proactively gathering the missing information that is crucial for high-quality solutions. In this work, we introduce a new task paradigm: proactive information gathering, where LLMs must identify gaps in the provided context and strategically elicit implicit user knowledge through targeted questions. To systematically study and train this capability, we design a scalable framework that generates partially specified, real-world tasks, masking key information and simulating authentic ambiguity. Within this setup, our core innovation is a reinforcement finetuning strategy that rewards questions that elicit genuinely new, implicit user information -- such as hidden domain expertise or fine-grained requirements -- that would otherwise remain unspoken. Experiments demonstrate that our trained Qwen-2.5-7B model significantly outperforms o3-mini by 18% on automatic evaluation metrics. More importantly, human evaluation reveals that clarification questions and final outlines generated by our model are favored by human annotators by 42% and 28% respectively. Together, these results highlight the value of proactive clarification in elevating LLMs from passive text generators to genuinely collaborative thought partners.