Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Environments with Delayed Rewards
This addresses delayed rewards in RL, which is a bottleneck for applications like robotics and games, though it appears incremental as it builds on existing PPO and TRPO frameworks.
The paper tackles the problem of delayed rewards in reinforcement learning by enhancing Proximal Policy Optimization with a hybrid policy architecture and Time Window Temporal Logic-based reward shaping, demonstrating improved learning speed and final performance on inverted pendulum and lunar lander environments compared to standard methods.
In this paper, we tackle the challenging problem of delayed rewards in reinforcement learning (RL). While Proximal Policy Optimization (PPO) has emerged as a leading Policy Gradient method, its performance can degrade under delayed rewards. We introduce two key enhancements to PPO: a hybrid policy architecture that combines an offline policy (trained on expert demonstrations) with an online PPO policy, and a reward shaping mechanism using Time Window Temporal Logic (TWTL). The hybrid architecture leverages offline data throughout training while maintaining PPO's theoretical guarantees. Building on the monotonic improvement framework of Trust Region Policy Optimization (TRPO), we prove that our approach ensures improvement over both the offline policy and previous iterations, with a bounded performance gap of $(2ςγα^2)/(1-γ)^2$, where $α$ is the mixing parameter, $γ$ is the discount factor, and $ς$ bounds the expected advantage. Additionally, we prove that our TWTL-based reward shaping preserves the optimal policy of the original problem. TWTL enables formal translation of temporal objectives into immediate feedback signals that guide learning. We demonstrate the effectiveness of our approach through extensive experiments on an inverted pendulum and a lunar lander environments, showing improvements in both learning speed and final performance compared to standard PPO and offline-only approaches.