Semantically Labelled Automata for Multi-Task Reinforcement Learning with LTL Instructions
This addresses the problem of generalizing to unseen tasks in reinforcement learning for domains requiring formal specifications, representing a novel method for a known bottleneck.
The paper tackles multi-task reinforcement learning with tasks specified as linear temporal logic (LTL) formulae by introducing a novel task embedding technique using semantically labelled automata, achieving state-of-the-art performance and scaling to complex specifications where existing methods fail.
We study multi-task reinforcement learning (RL), a setting in which an agent learns a single, universal policy capable of generalising to arbitrary, possibly unseen tasks. We consider tasks specified as linear temporal logic (LTL) formulae, which are commonly used in formal methods to specify properties of systems, and have recently been successfully adopted in RL. In this setting, we present a novel task embedding technique leveraging a new generation of semantic LTL-to-automata translations, originally developed for temporal synthesis. The resulting semantically labelled automata contain rich, structured information in each state that allow us to (i) compute the automaton efficiently on-the-fly, (ii) extract expressive task embeddings used to condition the policy, and (iii) naturally support full LTL. Experimental results in a variety of domains demonstrate that our approach achieves state-of-the-art performance and is able to scale to complex specifications where existing methods fail.