ELO-Rated Sequence Rewards: Advancing Reinforcement Learning Models
This addresses a key bottleneck in reinforcement learning for long-term tasks, offering a novel method to improve training without expert cardinal rewards.
The paper tackles the challenge of reward function design in long-term reinforcement learning by proposing ERRL, which uses expert preferences over trajectories to compute ELO ratings as rewards and includes a redistribution algorithm for stability. In tests with up to 5000 steps, it outperforms state-of-the-art baselines.
Reinforcement Learning (RL) heavily relies on the careful design of the reward function. However, accurately assigning rewards to each state-action pair in Long-Term Reinforcement Learning (LTRL) tasks remains a significant challenge. As a result, RL agents are often trained under expert guidance. Inspired by the ordinal utility theory in economics, we propose a novel reward estimation algorithm: ELO-Rating based Reinforcement Learning (ERRL). This approach features two key contributions. First, it uses expert preferences over trajectories rather than cardinal rewards (utilities) to compute the ELO rating of each trajectory as its reward. Second, a new reward redistribution algorithm is introduced to alleviate training instability in the absence of a fixed anchor reward. In long-term scenarios (up to 5000 steps), where traditional RL algorithms struggle, our method outperforms several state-of-the-art baselines. Additionally, we conduct a comprehensive analysis of how expert preferences influence the results.