When and What to Ask: AskBench and Rubric-Guided RLVR for LLM Clarification
For LLM developers and users, this work provides a benchmark and training method to reduce hallucinations from ambiguous or misleading prompts, though the approach is incremental.
The paper introduces AskBench, a benchmark for evaluating LLMs' ability to ask for clarification when prompts are ambiguous or contain false premises, and proposes rubric-guided RLVR to improve clarification behavior, achieving consistent gains in accuracy and efficiency.
Large language models (LLMs) often respond even when prompts omit critical details or include misleading information, leading to hallucinations or reinforced misconceptions. We study how to evaluate and improve LLMs' ability to decide when and what to ask for clarification without sacrificing task performance. We introduce AskBench, an interactive benchmark that converts standard QA pairs into multi-turn interactions with explicit checkpoints. A unified judge loop evaluates final answers and simulates user responses as needed. AskBench covers two settings: AskMind, with intent-deficient queries requiring clarification, and AskOverconfidence, with queries containing false premises that must be identified and corrected. We further propose rubric-guided reinforcement learning with verifier-based rewards (RLVR), which uses structured rubrics to encourage targeted clarification. Experiments show consistent improvements in accuracy, rubric adherence, and interaction efficiency, with strong generalization to unseen domains.