Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slow
This addresses efficiency problems for researchers and practitioners using RL in visual domains, though it appears incremental as it builds on existing off-policy algorithms.
The paper tackles the high computational demands of recurrent off-policy deep reinforcement learning in image-based settings by introducing RISE, which integrates recurrent networks without significant overhead, resulting in a 35.6% performance improvement on the Atari benchmark.
Recurrent off-policy deep reinforcement learning models achieve state-of-the-art performance but are often sidelined due to their high computational demands. In response, we introduce RISE (Recurrent Integration via Simplified Encodings), a novel approach that can leverage recurrent networks in any image-based off-policy RL setting without significant computational overheads via using both learnable and non-learnable encoder layers. When integrating RISE into leading non-recurrent off-policy RL algorithms, we observe a 35.6% human-normalized interquartile mean (IQM) performance improvement across the Atari benchmark. We analyze various implementation strategies to highlight the versatility and potential of our proposed framework.