Live LTL Progress Tracking: Towards Task-Based Exploration
This work addresses the challenge of non-Markovian objectives in reinforcement learning by providing a method to encode detailed task execution information, but it is currently a conceptual framework with no empirical validation.
The paper introduces Live LTL Progress Tracking, a framework that tracks agent progress through multi-stage tasks specified in linear temporal logic (LTL) using a 'tracking vector' that updates at each time step. The framework assigns true, false, or 'open' labels to task status, enabling potential applications in performance metrics, exploration, and reward shaping for reinforcement learning.
Motivated by the challenge presented by non-Markovian objectives in reinforcement learning (RL), we present a novel framework to track and represent the progress of autonomous agents through complex, multi-stage tasks. Given a specification in finite linear temporal logic (LTL), the framework establishes a 'tracking vector' which updates at each time step in a trajectory rollout. The values of the vector represent the status of the specification as the trajectory develops, assigning true, false, or 'open' labels (where 'open' is used for indeterminate cases). Applied to an LTL formula tree, the tracking vector can be used to encode detailed information about how a task is executed over a trajectory, providing a potential tool for new performance metrics, diverse exploration, and reward shaping. In this paper, we formally present the framework and algorithm, collectively named Live LTL Progress Tracking, give a simple working example, and demonstrate avenues for its integration into RL models. Future work will apply the framework to problems such as task-space exploration and diverse solution-finding in RL.