Iterated $Q$-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning
This addresses a bottleneck in reinforcement learning for researchers and practitioners by potentially enhancing learning algorithms, but it appears incremental as it builds on existing Bellman operator ideas.
The paper tackles the challenge of improving sample efficiency and performance in deep reinforcement learning by enabling multiple consecutive Bellman updates, introducing the iterated Q-Network (i-QN) method. The result is demonstrated empirically with advantages shown in Atari 2600 games and MuJoCo continuous control problems, though no specific numbers are provided.
The vast majority of Reinforcement Learning methods is largely impacted by the computation effort and data requirements needed to obtain effective estimates of action-value functions, which in turn determine the quality of the overall performance and the sample-efficiency of the learning procedure. Typically, action-value functions are estimated through an iterative scheme that alternates the application of an empirical approximation of the Bellman operator and a subsequent projection step onto a considered function space. It has been observed that this scheme can be potentially generalized to carry out multiple iterations of the Bellman operator at once, benefiting the underlying learning algorithm. However, till now, it has been challenging to effectively implement this idea, especially in high-dimensional problems. In this paper, we introduce iterated $Q$-Network (i-QN), a novel principled approach that enables multiple consecutive Bellman updates by learning a tailored sequence of action-value functions where each serves as the target for the next. We show that i-QN is theoretically grounded and that it can be seamlessly used in value-based and actor-critic methods. We empirically demonstrate the advantages of i-QN in Atari $2600$ games and MuJoCo continuous control problems.