Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoning
This addresses the issue of improving reasoning generalization in LLMs for complex tasks, though it appears incremental as it builds on existing causal perspectives and policy optimization methods.
The paper tackles the problem of LLMs' reward mechanisms being overly focused on final answer correctness, which can penalize sound reasoning with wrong answers and reward lucky guesses with flawed logic, affecting generalization. The result is a proposed method, Group Causal Counterfactual Policy Optimization, that improves reasoning generalization, as demonstrated by extensive experiments on diverse benchmarks.
Large language models (LLMs) excel at complex tasks with advances in reasoning capabilities. However, existing reward mechanisms remain tightly coupled to final correctness and pay little attention to the underlying reasoning process: trajectories with sound reasoning but wrong answers receive low credit, while lucky guesses with flawed logic may be highly rewarded, affecting reasoning generalization. From a causal perspective, we interpret multi-candidate reasoning for a fixed question as a family of counterfactual experiments with theoretical supports. Building on this, we propose Group Causal Counterfactual Policy Optimization to explicitly train LLMs to learn generalizable reasoning patterns. It proposes an episodic causal counterfactual reward that jointly captures (i) robustness, encouraging the answer distribution induced by a reasoning step to remain stable under counterfactual perturbations; and (ii) effectiveness, enforcing sufficient variability so that the learned reasoning strategy can transfer across questions. We then construct token-level advantages from this reward and optimize the policy, encouraging LLMs to favor reasoning patterns that are process-valid and counterfactually robust. Extensive experiments on diverse benchmarks demonstrate its advantages.