Zero-Shot Instruction Following in RL via Structured LTL Representations
This addresses the challenge of enabling RL agents to execute arbitrary structured tasks in complex, multi-event settings, representing an incremental advance over prior LTL-based approaches.
The paper tackles the problem of reinforcement learning agents following complex Linear Temporal Logic (LTL) instructions in environments with multiple interacting high-level events, and the result is a novel method that achieves improved performance in a chess-based environment.
Linear temporal logic (LTL) is a compelling framework for specifying complex, structured tasks for reinforcement learning (RL) agents. Recent work has shown that interpreting LTL instructions as finite automata, which can be seen as high-level programs monitoring task progress, enables learning a single generalist policy capable of executing arbitrary instructions at test time. However, existing approaches fall short in environments where multiple high-level events (i.e., atomic propositions) can be true at the same time and potentially interact in complicated ways. In this work, we propose a novel approach to learning a multi-task policy for following arbitrary LTL instructions that addresses this shortcoming. Our method conditions the policy on sequences of simple Boolean formulae, which directly align with transitions in the automaton, and are encoded via a graph neural network (GNN) to yield structured task representations. Experiments in a complex chess-based environment demonstrate the advantages of our approach.