Hierarchical Deep Q-Network from Imperfect Demonstrations in Minecraft
This work addresses the challenge of efficient reinforcement learning from suboptimal human demonstrations in complex environments like Minecraft, representing an incremental improvement over existing methods.
The paper tackles the problem of learning from imperfect demonstrations in Minecraft by introducing Hierarchical Deep Q-Network (HDQfD), which won first place in the MineRL competition by extracting meta-actions and subgoals from expert trajectories and using a structured replay buffer to filter out poor-quality data.
We present Hierarchical Deep Q-Network (HDQfD) that took first place in the MineRL competition. HDQfD works on imperfect demonstrations and utilizes the hierarchical structure of expert trajectories. We introduce the procedure of extracting an effective sequence of meta-actions and subgoals from demonstration data. We present a structured task-dependent replay buffer and adaptive prioritizing technique that allow the HDQfD agent to gradually erase poor-quality expert data from the buffer. In this paper, we present the details of the HDQfD algorithm and give the experimental results in the Minecraft domain.