TreeAdv: Tree-Structured Advantage Redistribution for Group-Based RL
This addresses a specific bottleneck in reinforcement learning for language model alignment, offering an incremental improvement for more efficient training on complex reasoning tasks.
The paper tackled the problem of sample inefficiency and length bias in group-based reinforcement learning for aligning large language models on reasoning tasks by introducing TreeAdv, which uses tree-structured advantage redistribution, resulting in outperforming GRPO and GSPO across 10 math reasoning benchmarks with substantially fewer generated tokens.
Reinforcement learning with group-based objectives, such as Group Relative Policy Optimization (GRPO), is a common framework for aligning large language models on complex reasoning tasks. However, standard GRPO treats each rollout trajectory as an independent flat sequence and assigns a single sequence-level advantage to all tokens, which leads to sample inefficiency and a length bias toward verbose, redundant chains of thought without improving logical depth. We introduce TreeAdv (Tree-Structured Advantage Redistribution for Group-Based RL), which makes the tree structure of group rollouts explicit for both exploration and advantage assignment. Specifically, TreeAdv builds a group of trees (a forest) based on an entropy-driven sampling method where each tree branches at high-uncertainty decisions while sharing low-uncertainty tokens across rollouts. Then, TreeAdv aggregates token-level advantages for internal tree segments by redistributing the advantages of complete rollouts (all leaf nodes), and TreeAdv can easily apply to group-based objectives such as GRPO or GSPO. Across 10 math reasoning benchmarks, TreeAdv consistently outperforms GRPO and GSPO, while using substantially fewer generated tokens under identical supervision, data, and decoding budgets.