Confounding Robust Continuous Control via Automatic Reward Shaping
This work addresses the challenge of accelerating RL training in complex continuous control tasks, offering a robust solution against confounding effects, though it is an incremental step in applying causal methods to this domain.
The paper tackles the problem of designing effective reward shaping functions for continuous control in reinforcement learning, particularly under unobserved confounding variables, by proposing an automatic method that learns from offline datasets and demonstrates strong performance guarantees on benchmarks.
Reward shaping has been applied widely to accelerate Reinforcement Learning (RL) agents' training. However, a principled way of designing effective reward shaping functions, especially for complex continuous control problems, remains largely under-explained. In this work, we propose to automatically learn a reward shaping function for continuous control problems from offline datasets, potentially contaminated by unobserved confounding variables. Specifically, our method builds upon the recently proposed causal Bellman equation to learn a tight upper bound on the optimal state values, which is then used as the potentials in the Potential-Based Reward Shaping (PBRS) framework. Our proposed reward shaping algorithm is tested with Soft-Actor-Critic (SAC) on multiple commonly used continuous control benchmarks and exhibits strong performance guarantees under unobserved confounders. More broadly, our work marks a solid first step towards confounding robust continuous control from a causal perspective. Code for training our reward shaping functions can be found at https://github.com/mateojuliani/confounding_robust_cont_control.