Counterfactual Self-Questioning for Stable Policy Optimization in Language Models
This addresses the challenge of scalable and stable self-improvement in language models for tasks like mathematical reasoning, though it is incremental as it builds on existing self-improvement approaches.
The paper tackles the problem of training instability and complexity in language model self-improvement by proposing Counterfactual Self-Questioning, a framework where a single model generates and evaluates counterfactual critiques of its own reasoning, resulting in improved accuracy and training stability on mathematical reasoning benchmarks, especially for smaller models.
Recent work on language model self-improvement shows that models can refine their own reasoning through reflection, verification, debate, or self-generated rewards. However, most existing approaches rely on external critics, learned reward models, or ensemble sampling, which increases complexity and training instability. We propose Counterfactual Self-Questioning, a framework in which a single language model generates and evaluates counterfactual critiques of its own reasoning. The method produces an initial reasoning trace, formulates targeted questions that challenge potential failure points, and generates alternative reasoning trajectories that expose incorrect assumptions or invalid steps. These counterfactual trajectories provide structured relative feedback that can be directly used for policy optimization without auxiliary models. Experiments on multiple mathematical reasoning benchmarks show that counterfactual self-questioning improves accuracy and training stability, particularly for smaller models, enabling scalable self-improvement using internally generated supervision alone.