Using Contrastive Samples for Identifying and Leveraging Possible Causal Relationships in Reinforcement Learning
This addresses the problem of causal attribution in reinforcement learning for researchers and practitioners, offering an incremental improvement over existing methods.
The paper tackles the challenge of linking actions to long-term rewards in reinforcement learning by identifying transitions with significant state deviations and reward variations as possible causal effects, and using contrastive samples to improve training; the proposed Contrastive Experience Replay (CER) method outperforms standard value-based methods on 2D navigation tasks.
A significant challenge in reinforcement learning is quantifying the complex relationship between actions and long-term rewards. The effects may manifest themselves over a long sequence of state-action pairs, making them hard to pinpoint. In this paper, we propose a method to link transitions with significant deviations in state with unusually large variations in subsequent rewards. Such transitions are marked as possible causal effects, and the corresponding state-action pairs are added to a separate replay buffer. In addition, we include \textit{contrastive} samples corresponding to transitions from a similar state but with differing actions. Including this Contrastive Experience Replay (CER) during training is shown to outperform standard value-based methods on 2D navigation tasks. We believe that CER can be useful for a broad class of learning tasks, including for any off-policy reinforcement learning algorithm.