Causal Influence Detection for Improving Efficiency in Reinforcement Learning
This work addresses data efficiency for reinforcement learning agents in environments with independent entities, though it is incremental as it builds on existing RL methods with a novel integration.
The paper tackles the problem of inefficient learning in reinforcement learning environments with sparse interactions by introducing a situation-dependent causal influence measure based on conditional mutual information to guide learning, resulting in strong increases in data efficiency on robotic manipulation tasks.
Many reinforcement learning (RL) environments consist of independent entities that interact sparsely. In such environments, RL agents have only limited influence over other entities in any particular situation. Our idea in this work is that learning can be efficiently guided by knowing when and what the agent can influence with its actions. To achieve this, we introduce a measure of \emph{situation-dependent causal influence} based on conditional mutual information and show that it can reliably detect states of influence. We then propose several ways to integrate this measure into RL algorithms to improve exploration and off-policy learning. All modified algorithms show strong increases in data efficiency on robotic manipulation tasks.