LTL-Constrained Policy Optimization with Cycle Experience Replay
This work addresses the challenge of ensuring LTL constraint satisfaction in deep reinforcement learning for continuous control, which is incremental as it builds on existing reward-shaping techniques.
The paper tackles the problem of optimizing reinforcement learning policies under Linear Temporal Logic (LTL) constraints, which is difficult due to sparse satisfaction signals, and introduces Cycle Experience Replay (CyclER) to address this, showing it outperforms existing methods in continuous control domains.
Linear Temporal Logic (LTL) offers a precise means for constraining the behavior of reinforcement learning agents. However, in many settings where both satisfaction and optimality conditions are present, LTL is insufficient to capture both. Instead, LTL-constrained policy optimization, where the goal is to optimize a scalar reward under LTL constraints, is needed. This constrained optimization problem proves difficult in deep Reinforcement Learning (DRL) settings, where learned policies often ignore the LTL constraint due to the sparse nature of LTL satisfaction. To alleviate the sparsity issue, we introduce Cycle Experience Replay (CyclER), a novel reward shaping technique that exploits the underlying structure of the LTL constraint to guide a policy towards satisfaction by encouraging partial behaviors compliant with the constraint. We provide a theoretical guarantee that optimizing CyclER will achieve policies that satisfy the LTL constraint with near-optimal probability. We evaluate CyclER in three continuous control domains. Our experimental results show that optimizing CyclER in tandem with the existing scalar reward outperforms existing reward-shaping methods at finding performant LTL-satisfying policies.