RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models
This addresses the challenge of costly and inflexible reward modeling in multi-task reinforcement learning for LLMs, offering a dynamic solution that improves generalization.
The paper tackles the problem of poor generalization in static reward models for large language model alignment by introducing RLAR, an agent-driven framework that dynamically assigns tailored reward functions, resulting in performance gains of 10 to 60 across multiple tasks and outperforming static baselines on RewardBench-V2.
Large language model alignment via reinforcement learning depends critically on reward function quality. However, static, domain-specific reward models are often costly to train and exhibit poor generalization in out-of-distribution scenarios encountered during RL iterations. We present RLAR (Reinforcement Learning from Agent Rewards), an agent-driven framework that dynamically assigns tailored reward functions to individual queries. Specifically, RLAR transforms reward acquisition into a dynamic tool synthesis and invocation task. It leverages LLM agents to autonomously retrieve optimal reward models from the Internet and synthesize programmatic verifiers through code generation. This allows the reward system to self-evolve with the shifting data distributions during training. Experimental results demonstrate that RLAR yields consistent performance gains ranging from 10 to 60 across mathematics, coding, translation, and dialogue tasks. On RewardBench-V2, RLAR significantly outperforms static baselines and approaches the performance upper bound, demonstrating superior generalization through dynamic reward orchestration. The data and code are available on this link: https://github.com/ZhuoerFeng/RLAR.