Hierarchical reinforcement learning with natural language subgoals
This work addresses the problem of enabling goal-directed behavior in complex, open-ended environments for AI systems, representing an incremental improvement by combining human supervision with reinforcement learning.
The paper tackles the challenge of implementing hierarchical reinforcement learning in realistic environments by using natural language data from humans to supervise the sub-goal space, resulting in an approach that outperforms expert behavior cloning and HRL from scratch.
Hierarchical reinforcement learning has been a compelling approach for achieving goal directed behavior over long sequences of actions. However, it has been challenging to implement in realistic or open-ended environments. A main challenge has been to find the right space of sub-goals over which to instantiate a hierarchy. We present a novel approach where we use data from humans solving these tasks to softly supervise the goal space for a set of long range tasks in a 3D embodied environment. In particular, we use unconstrained natural language to parameterize this space. This has two advantages: first, it is easy to generate this data from naive human participants; second, it is flexible enough to represent a vast range of sub-goals in human-relevant tasks. Our approach outperforms agents that clone expert behavior on these tasks, as well as HRL from scratch without this supervised sub-goal space. Our work presents a novel approach to combining human expert supervision with the benefits and flexibility of reinforcement learning.