Learning Nonlinear Causal Reductions to Explain Reinforcement Learning Policies
This work addresses the challenge of interpretability in complex RL systems for researchers and practitioners, though it appears incremental as it builds on causal perspectives with a novel reduction method.
The paper tackled the problem of explaining why reinforcement learning policies succeed or fail by developing a nonlinear Causal Model Reduction framework that learns simplified high-level causal models from perturbed actions, demonstrating its ability to uncover behavioral patterns, biases, and failure modes in tasks like pendulum control and robot table tennis.
Why do reinforcement learning (RL) policies fail or succeed? This is a challenging question due to the complex, high-dimensional nature of agent-environment interactions. In this work, we take a causal perspective on explaining the behavior of RL policies by viewing the states, actions, and rewards as variables in a low-level causal model. We introduce random perturbations to policy actions during execution and observe their effects on the cumulative reward, learning a simplified high-level causal model that explains these relationships. To this end, we develop a nonlinear Causal Model Reduction framework that ensures approximate interventional consistency, meaning the simplified high-level model responds to interventions in a similar way as the original complex system. We prove that for a class of nonlinear causal models, there exists a unique solution that achieves exact interventional consistency, ensuring learned explanations reflect meaningful causal patterns. Experiments on both synthetic causal models and practical RL tasks-including pendulum control and robot table tennis-demonstrate that our approach can uncover important behavioral patterns, biases, and failure modes in trained RL policies.