Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL
For developers of LLMs, this work provides a method to teach models to abstain and clarify missing information through verifiable rewards, reducing hallucination without sacrificing performance on answerable queries.
The paper addresses the problem of LLMs hallucinating on unanswerable queries and proposes Abstain-R1, a 3B model trained with a clarification-aware RLVR reward that jointly optimizes abstention and post-refusal clarification. Abstain-R1 achieves competitive performance with larger systems like DeepSeek-R1 on abstention benchmarks while maintaining strong performance on answerable queries.
Reinforcement fine-tuning improves the reasoning ability of large language models, but it can also encourage them to answer unanswerable queries by guessing or hallucinating missing information. Existing abstention methods either train models to produce generic refusals or encourage follow-up clarifications without verifying whether those clarifications identify the key missing information. We study queries that are clear in meaning but cannot be reliably resolved from the given information, and argue that a reliable model should not only abstain, but also explain what is missing. We propose a clarification-aware RLVR reward that, while rewarding correct answers on answerable queries, jointly optimizes explicit abstention and semantically aligned post-refusal clarification on unanswerable queries. Using this reward, we train Abstain-R1, a 3B model that improves abstention and clarification on unanswerable queries while preserving strong performance on answerable ones. Experiments on Abstain-Test, Abstain-QA, and SelfAware show that Abstain-R1 substantially improves over its base model and achieves unanswerable-query behavior competitive with larger systems including DeepSeek-R1, suggesting that calibrated abstention and clarification can be learned through verifiable rewards rather than emerging from scale alone.