Hierarchical Policy Search via Return-Weighted Density Estimation
This addresses the challenge of hierarchical reinforcement learning in complex real-world problems like robotics, though it appears incremental as it builds on existing HRL methods to improve mode identification.
The paper tackles the problem of learning optimal policies from multi-modal reward functions in reinforcement learning by proposing HPSDE, a method that efficiently identifies reward modes via return-weighted density estimation and automatically determines option policies, successfully applying it to a robotic manipulator motion planning task.
Learning an optimal policy from a multi-modal reward function is a challenging problem in reinforcement learning (RL). Hierarchical RL (HRL) tackles this problem by learning a hierarchical policy, where multiple option policies are in charge of different strategies corresponding to modes of a reward function and a gating policy selects the best option for a given context. Although HRL has been demonstrated to be promising, current state-of-the-art methods cannot still perform well in complex real-world problems due to the difficulty of identifying modes of the reward function. In this paper, we propose a novel method called hierarchical policy search via return-weighted density estimation (HPSDE), which can efficiently identify the modes through density estimation with return-weighted importance sampling. Our proposed method finds option policies corresponding to the modes of the return function and automatically determines the number and the location of option policies, which significantly reduces the burden of hyper-parameters tuning. Through experiments, we demonstrate that the proposed HPSDE successfully learns option policies corresponding to modes of the return function and that it can be successfully applied to a challenging motion planning problem of a redundant robotic manipulator.