Hierarchical Reinforcement Learning with Targeted Causal Interventions
This work addresses the problem of training efficiency in HRL for long-horizon tasks with sparse rewards, representing an incremental advance by integrating causal modeling into existing HRL frameworks.
The paper tackles the challenge of efficiently discovering hierarchical structures in hierarchical reinforcement learning (HRL) by modeling subgoals as a causal graph and using targeted interventions based on causal importance, resulting in significant improvements in training cost, with notable gains on tree structures and Erdős-Rényi random graphs.
Hierarchical reinforcement learning (HRL) improves the efficiency of long-horizon reinforcement-learning tasks with sparse rewards by decomposing the task into a hierarchy of subgoals. The main challenge of HRL is efficient discovery of the hierarchical structure among subgoals and utilizing this structure to achieve the final goal. We address this challenge by modeling the subgoal structure as a causal graph and propose a causal discovery algorithm to learn it. Additionally, rather than intervening on the subgoals at random during exploration, we harness the discovered causal model to prioritize subgoal interventions based on their importance in attaining the final goal. These targeted interventions result in a significantly more efficient policy in terms of the training cost. Unlike previous work on causal HRL, which lacked theoretical analysis, we provide a formal analysis of the problem. Specifically, for tree structures and, for a variant of Erdős-Rényi random graphs, our approach results in remarkable improvements. Our experimental results on HRL tasks also illustrate that our proposed framework outperforms existing work in terms of training cost.