Generalizing Verifiable Instruction Following
This addresses the challenge of making AI models reliably follow diverse human instructions, which is crucial for practical human-AI interaction, though it is incremental as it builds on existing methods for constraint verification.
The paper tackles the problem of language models overfitting to a small set of verifiable constraints in precise instruction following, introducing IFBench with 58 new constraints to evaluate generalization, and shows that reinforcement learning with verifiable rewards (RLVR) significantly improves performance.
A crucial factor for successful human and AI interaction is the ability of language models or chatbots to follow human instructions precisely. A common feature of instructions are output constraints like ``only answer with yes or no" or ``mention the word `abrakadabra' at least 3 times" that the user adds to craft a more useful answer. Even today's strongest models struggle with fulfilling such constraints. We find that most models strongly overfit on a small set of verifiable constraints from the benchmarks that test these abilities, a skill called precise instruction following, and are not able to generalize well to unseen output constraints. We introduce a new benchmark, IFBench, to evaluate precise instruction following generalization on 58 new, diverse, and challenging verifiable out-of-domain constraints. In addition, we perform an extensive analysis of how and on what data models can be trained to improve precise instruction following generalization. Specifically, we carefully design constraint verification modules and show that reinforcement learning with verifiable rewards (RLVR) significantly improves instruction following. In addition to IFBench, we release 29 additional new hand-annotated training constraints and verification functions, RLVR training prompts, and code.