Learning to Follow Language Instructions with Compositional Policies
This addresses the challenge of sample efficiency in reinforcement learning for language-guided agents, though it is incremental as it builds on existing compositional and language representation methods.
The paper tackles the problem of learning to execute natural language instructions in goal-reaching tasks by leveraging compositionality to reduce sample complexity, achieving an 86% reduction in training steps for a second task after mastering a first one.
We propose a framework that learns to execute natural language instructions in an environment consisting of goal-reaching tasks that share components of their task descriptions. Our approach leverages the compositionality of both value functions and language, with the aim of reducing the sample complexity of learning novel tasks. First, we train a reinforcement learning agent to learn value functions that can be subsequently composed through a Boolean algebra to solve novel tasks. Second, we fine-tune a seq2seq model pretrained on web-scale corpora to map language to logical expressions that specify the required value function compositions. Evaluating our agent in the BabyAI domain, we observe a decrease of 86% in the number of training steps needed to learn a second task after mastering a single task. Results from ablation studies further indicate that it is the combination of compositional value functions and language representations that allows the agent to quickly generalize to new tasks.