Blockwise Advantage Estimation for Multi-Objective RL with Verifiable Rewards
This addresses the challenge of optimizing multiple objectives in sequential text generation for applications like math tasks, though it is incremental as it builds on existing GRPO methods.
The paper tackled the problem of objective interference and misattributed credit in multi-objective reinforcement learning for structured text generation by proposing Blockwise Advantage Estimation, which assigns separate advantages to each objective and applies them only to relevant text blocks, resulting in competitive performance with state-of-the-art methods and preserved test-time gains on math tasks.
Group Relative Policy Optimization (GRPO) assigns a single scalar advantage to all tokens in a completion. For structured generations with explicit segments and objectives, this couples unrelated reward signals across segments, leading to objective interference and misattributed credit. We propose Blockwise Advantage Estimation, a family of GRPO-compatible methods that assigns each objective its own advantage and applies it only to the tokens in the corresponding text block, reducing reliance on hand-designed scalar rewards and scaling naturally to additional objectives. A key challenge is estimating advantages for later blocks whose rewards are conditioned on sampled prefixes; standard unbiased approaches require expensive nested rollouts from intermediate states. Concretely, we introduce an Outcome-Conditioned Baseline that approximates intermediate state values using only within-group statistics by stratifying samples according to a prefix-derived intermediate outcome. On math tasks with uncertainty estimation, our method mitigates reward interference, is competitive with a state-of-the-art reward-designed approach, and preserves test-time gains from confidence-weighted ensembling. More broadly, it provides a modular recipe for optimizing sequential objectives in structured generations without additional rollouts.