Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning
This addresses the bottleneck of requiring human-annotated sub-goals for hierarchical RL, making it more practical for long-horizon sparse reward tasks.
The paper tackled the problem of sample inefficiency in hierarchical reinforcement learning by automatically extracting sub-goals from human eye gaze instead of using manual annotations, resulting in a significantly more sample-efficient agent on the challenging Montezuma's Revenge task.
While deep reinforcement learning (RL) agents outperform humans on an increasing number of tasks, training them requires data equivalent to decades of human gameplay. Recent hierarchical RL methods have increased sample efficiency by incorporating information inherent to the structure of the decision problem but at the cost of having to discover or use human-annotated sub-goals that guide the learning process. We show that intentions of human players, i.e. the precursor of goal-oriented decisions, can be robustly predicted from eye gaze even for the long-horizon sparse rewards task of Montezuma's Revenge - one of the most challenging RL tasks in the Atari2600 game suite. We propose Int-HRL: Hierarchical RL with intention-based sub-goals that are inferred from human eye gaze. Our novel sub-goal extraction pipeline is fully automatic and replaces the need for manual sub-goal annotation by human experts. Our evaluations show that replacing hand-crafted sub-goals with automatically extracted intentions leads to a HRL agent that is significantly more sample efficient than previous methods.