Variable-Agnostic Causal Exploration for Reinforcement Learning
This addresses the challenge of exploration inefficiency in RL for agents operating in complex environments, representing a novel method rather than an incremental improvement.
The paper tackles the problem of inefficient exploration in reinforcement learning due to real-world cause-and-effect dynamics by introducing VACERL, a framework that uses causal relationships without specifying environmental causal variables, resulting in significant performance improvements in grid-world, 2D games, and robotic domains, especially in sparse-reward and noisy-action scenarios.
Modern reinforcement learning (RL) struggles to capture real-world cause-and-effect dynamics, leading to inefficient exploration due to extensive trial-and-error actions. While recent efforts to improve agent exploration have leveraged causal discovery, they often make unrealistic assumptions of causal variables in the environments. In this paper, we introduce a novel framework, Variable-Agnostic Causal Exploration for Reinforcement Learning (VACERL), incorporating causal relationships to drive exploration in RL without specifying environmental causal variables. Our approach automatically identifies crucial observation-action steps associated with key variables using attention mechanisms. Subsequently, it constructs the causal graph connecting these steps, which guides the agent towards observation-action pairs with greater causal influence on task completion. This can be leveraged to generate intrinsic rewards or establish a hierarchy of subgoals to enhance exploration efficiency. Experimental results showcase a significant improvement in agent performance in grid-world, 2d games and robotic domains, particularly in scenarios with sparse rewards and noisy actions, such as the notorious Noisy-TV environments.