Integrating Causal DAGs in Deep RL: Activating Minimal Markovian States with Multi-Order Exposure
For researchers in causal RL, this work identifies a key limitation of minimal state representations and provides a practical solution, though the findings are incremental.
The paper addresses the problem of constructing minimal Markovian states from causal graphs for deep RL, finding that minimal representations alone fail to improve performance. They propose MOSE, which feeds multi-order historical states into the Q-function, consistently outperforming baselines on benchmarks.
Online reinforcement learning (RL) relies on the Markov property for guaranteed performance, but real-world applications often lack well-defined states given raw observed variables. While causal RL has attracted growing interest, existing work typically assumes Markovian states are provided and focuses on using causality to accelerate learning, leaving a fundamental gap: \emph{given a longitudinal causal graph over observed variables, how does one construct MDP states that provably satisfy the Markov property?} We address this by providing a procedure that constructs a provably minimal state representation. In deep RL, we observe that the minimal representation alone empirically fails to improve performance, indicating that neural networks cannot directly exploit Markovian minimality. To address this, we propose \textbf{MOSE} (Multi-Order State Exposure), which feeds multi-order historical state constructions into the same $Q$-function. MOSE consistently outperforms both the minimal state construction and single-window policies on common benchmarks and synthetic datasets. Including the minimal representation alongside MOSE can further improve performance. Our results establish a core principle for causal deep RL: minimal sufficiency is not enough, and \emph{controlled redundancy} is necessary to unlock the benefit of causal state information.