Multi-Objective and Mixed-Reward Reinforcement Learning via Reward-Decorrelated Policy Optimization
For practitioners training large language models with complex reward structures, RDPO offers a practical method to improve multi-task performance without sacrificing general capabilities.
RDPO improves multi-objective and mixed-reward RL by stabilizing advantage allocation and reducing reward correlation, enhancing instruction following, writing quality, and robustness in post-training of LongCat-Flash while remaining competitive on reasoning and coding.
Complex reinforcement learning environments frequently employ multi-task and mixed-reward formulations. In these settings, heterogeneous reward distributions and correlated reward dimensions often destabilize the construction of scalar advantages. To address these challenges, we propose Reward-Decorrelated Policy Optimization (RDPO), a reward-processing method designed to explicitly target both failure modes. RDPO first utilizes Magnitude-Aware Quantile normalization to stabilize prompt-level advantage allocation across binary, fractional, and continuous rewards. It then applies Mahalanobis whitening within each active reward subspace to mitigate correlation redundancy prior to aggregation. When applied during the post-training of LongCat-Flash, RDPO enhances instruction following, writing quality, and robustness to hard prompts while remaining broadly competitive on reasoning and coding evaluations.