Hindsight Trust Region Policy Optimization
This addresses the problem of sparse rewards in reinforcement learning for domains like robotics and games, though it appears incremental as an extension of TRPO.
The paper tackled the challenge of reinforcement learning with sparse rewards by proposing Hindsight Trust Region Policy Optimization (HTRPO), which extended TRPO with hindsight techniques and introduced QKL and Hindsight Goal Filtering, resulting in consistent outperformance over TRPO and a state-of-the-art algorithm in various tasks.
Reinforcement Learning(RL) with sparse rewards is a major challenge. We propose \emph{Hindsight Trust Region Policy Optimization}(HTRPO), a new RL algorithm that extends the highly successful TRPO algorithm with \emph{hindsight} to tackle the challenge of sparse rewards. Hindsight refers to the algorithm's ability to learn from information across goals, including ones not intended for the current task. HTRPO leverages two main ideas. It introduces QKL, a quadratic approximation to the KL divergence constraint on the trust region, leading to reduced variance in KL divergence estimation and improved stability in policy update. It also presents Hindsight Goal Filtering(HGF) to select conductive hindsight goals. In experiments, we evaluate HTRPO in various sparse reward tasks, including simple benchmarks, image-based Atari games, and simulated robot control. Ablation studies indicate that QKL and HGF contribute greatly to learning stability and high performance. Comparison results show that in all tasks, HTRPO consistently outperforms both TRPO and HPG, a state-of-the-art algorithm for RL with sparse rewards.