Policy Synthesis and Reinforcement Learning for Discounted LTL
This work addresses the problem of reward specification in reinforcement learning for researchers and practitioners, offering a method to improve robustness in policy synthesis, though it appears incremental by building on existing LTL and discounting techniques.
The paper tackles the challenge of manually specifying reward functions in reinforcement learning by using discounted linear temporal logic (LTL) to express objectives, showing that this approach reduces sensitivity to transition probability perturbations while maintaining expressivity, and demonstrates a reduction to discounted-sum reward via a reward machine for identical discount factors.
The difficulty of manually specifying reward functions has led to an interest in using linear temporal logic (LTL) to express objectives for reinforcement learning (RL). However, LTL has the downside that it is sensitive to small perturbations in the transition probabilities, which prevents probably approximately correct (PAC) learning without additional assumptions. Time discounting provides a way of removing this sensitivity, while retaining the high expressivity of the logic. We study the use of discounted LTL for policy synthesis in Markov decision processes with unknown transition probabilities, and show how to reduce discounted LTL to discounted-sum reward via a reward machine when all discount factors are identical.