From Atomic to Composite: Reinforcement Learning Enables Generalization in Complementary Reasoning
This work addresses the debate on RL's role in reasoning for AI researchers, offering insights into scalable generalization methods for complex tasks, though it is incremental in building on existing SFT and RL techniques.
The study tackled the problem of how reinforcement learning (RL) contributes to reasoning capabilities by investigating Complementary Reasoning, a task requiring integration of internal and external knowledge, using a synthetic dataset of human biographies. The results showed that RL acts as a reasoning synthesizer, enabling generalization in out-of-distribution settings, but only if the model first masters atomic skills via supervised fine-tuning (SFT), with SFT achieving near-perfect in-distribution accuracy but collapsing on zero-shot generalization.
The mechanism by which RL contributes to reasoning capabilities-whether it incentivizes the synthesis of new skills or merely amplifies existing behaviors-remains a subject of intense debate. In this work, we investigate this question through the lens of Complementary Reasoning, a complex task that requires integrating internal parametric knowledge with external contextual information. Using a controlled synthetic dataset of human biographies, we strictly decouple this ability into two atomic skills: Parametric Reasoning (relying on internal knowledge) and Contextual Reasoning (depending on external information). To rigorously assess capability boundaries, we evaluate generalization across three distinct levels of difficulty: I.I.D., Composition, and Zero-shot settings. We find that while SFT is sufficient for in-distribution performance, it struggles with O.O.D. generalization, particularly in Zero-shot settings where relational combinations are novel. Crucially, we identify the SFT Generalization Paradox: Models supervised solely on the composite task achieve near-perfect in-distribution accuracy but collapse on out-of-distribution generalization, indicating their reliance on rote memorization of path shortcuts. In contrast, we find that RL acts as a reasoning synthesizer rather than a probability amplifier. However, we uncover a strict atomic prerequisite: RL can only synthesize these complex strategies if the base model has first mastered the independent atomic skills (Parametric and Contextual) via SFT. These findings challenge the view of RL as a mere amplifier, suggesting that given sufficient atomic foundations, RL can actively synthesize complex reasoning strategies from learned primitives without explicit supervision on such complex strategies. This indicates that decoupled atomic training followed by RL offers a scalable path to generalization for complex reasoning tasks.