TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization
This addresses inefficiencies in search-augmented reasoning for complex tasks, though it appears incremental as it builds on existing RL frameworks like GRPO.
The paper tackled the problem of multi-turn tool-integrated reasoning in Large Language Models, where current reinforcement learning frameworks suffer from a 'Double Homogenization Dilemma' due to sparse outcome-level rewards, and the result was that their proposed TSPO method achieved average performance gains of 24% and 13.6% on Qwen2.5-3B and 7B models, respectively.
Multi-turn tool-integrated reasoning enables Large Language Models (LLMs) to solve complex tasks through iterative information retrieval. However, current reinforcement learning (RL) frameworks for search-augmented reasoning predominantly rely on sparse outcome-level rewards, leading to a "Double Homogenization Dilemma." This manifests as (1) Process homogenization, where the thinking, reasoning, and tooling involved in generation are ignored. (2) Intra-group homogenization, coarse-grained outcome rewards often lead to inefficiencies in intra-group advantage estimation with methods like Group Relative Policy Optimization (GRPO) during sampling. To address this, we propose Turn-level Stage-aware Policy Optimization (TSPO). TSPO introduces the First-Occurrence Latent Reward (FOLR) mechanism, allocating partial rewards to the step where the ground-truth answer first appears, thereby preserving process-level signals and increasing reward variance within groups without requiring external reward models or any annotations. Extensive experiments demonstrate that TSPO significantly outperforms state-of-the-art baselines, achieving average performance gains of 24% and 13.6% on Qwen2.5-3B and 7B models, respectively.