Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
This addresses the challenge of task generalization in RL for agents, though it is incremental as it builds on existing multi-task and hierarchical RL methods.
The paper tackles the problem of zero-shot task generalization in reinforcement learning by enabling agents to execute sequences of instructions after learning subtask skills, achieving generalization to unseen instructions and longer sequences through analogies and a hierarchical architecture.
As a step towards developing zero-shot task generalization capabilities in reinforcement learning (RL), we introduce a new RL problem where the agent should learn to execute sequences of instructions after learning useful skills that solve subtasks. In this problem, we consider two types of generalizations: to previously unseen instructions and to longer sequences of instructions. For generalization over unseen instructions, we propose a new objective which encourages learning correspondences between similar subtasks by making analogies. For generalization over sequential instructions, we present a hierarchical architecture where a meta controller learns to use the acquired skills for executing the instructions. To deal with delayed reward, we propose a new neural architecture in the meta controller that learns when to update the subtask, which makes learning more efficient. Experimental results on a stochastic 3D domain show that the proposed ideas are crucial for generalization to longer instructions as well as unseen instructions.