LEAF: Growing Trees Without Branching for Speech-Aware Large Language Model Post-Training
For researchers in speech-language model alignment, LEAF offers a more efficient RL method that achieves SOTA results with smaller models, addressing the coarse credit assignment bottleneck in GRPO-style methods.
LEAF improves speech-aware LLM post-training by replacing coarse token-level credit assignment with span-level advantages derived from shared prefixes in rollout batches, outperforming GRPO on speech QA and translation benchmarks while using fewer parameters.
State-of-the-art GRPO-style methods for speech-aware large language model post-training suffer from coarse credit assignment, broadcasting the same terminal-reward advantage to every token in a response. This ignores useful structure within rollout batches, where speech-conditioned completions often share prefixes before diverging at important decisions. We propose Low-rank Exploration with Adaptive Forking (LEAF), a retrospective tree-based RL method that recovers this structure without online branching or additional decoding. LEAF samples complete responses, selects high-surprisal boundaries, groups responses by shared prefixes, and assigns span-level advantages using descendant rewards. We theoretically justify LEAF's span-level credit assignment and boundary-selection design. Empirically, LEAF improves over GRPO across speech question answering and speech translation benchmarks under the same rollout and low-rank adaptation budget. Notably, smaller LEAF-trained models outperform current state-of-the-art, full-parameter baselines.