Simultaneously Updating All Persistence Values in Reinforcement Learning
This addresses the problem of inefficient exploration and vanishing action advantages in RL for agents operating at varying time scales, though it appears incremental as an extension of classic methods like Q-learning and DQN.
The paper tackles the sensitivity of reinforcement learning agents to time discretization by introducing the All-Persistence Bellman Operator, which simultaneously updates action values across different persistence levels, and demonstrates improved performance in tabular and Atari game experiments.
In reinforcement learning, the performance of learning agents is highly sensitive to the choice of time discretization. Agents acting at high frequencies have the best control opportunities, along with some drawbacks, such as possible inefficient exploration and vanishing of the action advantages. The repetition of the actions, i.e., action persistence, comes into help, as it allows the agent to visit wider regions of the state space and improve the estimation of the action effects. In this work, we derive a novel All-Persistence Bellman Operator, which allows an effective use of both the low-persistence experience, by decomposition into sub-transition, and the high-persistence experience, thanks to the introduction of a suitable bootstrap procedure. In this way, we employ transitions collected at any time scale to update simultaneously the action values of the considered persistence set. We prove the contraction property of the All-Persistence Bellman Operator and, based on it, we extend classic Q-learning and DQN. After providing a study on the effects of persistence, we experimentally evaluate our approach in both tabular contexts and more challenging frameworks, including some Atari games.