Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?
This addresses the generalizability problem for AI researchers and practitioners using RPT, showing it is incremental by highlighting limitations in domain transfer.
The paper investigates whether performance improvements from reinforcement post training (RPT) in large language models transfer to new domains, finding that gains are substantial on similar tasks but inconsistent or vanish on domains with different reasoning patterns.
Reinforcement post training (RPT) has recently shown promise in improving the reasoning abilities of large language models (LLMs). However, it remains unclear how well these improvements generalize to new domains, as prior work evaluates RPT models on data from the same domains used for fine-tuning. To understand the generalizability of RPT, we conduct two studies. (1) Observational: We compare a wide range of open-weight RPT models against their corresponding base models across multiple domains, including both seen and unseen domains in their fine-tuning data. (2) Interventional: we fine-tune LLMs with RPT on single domains and evaluate their performance across multiple domains. Both studies converge on the same conclusion that, although RPT brings substantial gains on tasks similar to the fine-tuning data, the gains generalize inconsistently and can vanish on domains with different reasoning patterns.