Causal Flow Q-Learning for Robust Offline Reinforcement Learning
This addresses robustness issues in offline RL for pixel-based tasks, offering a solution to confounding biases that affect policy learning from demonstrations.
The paper tackles the problem of unmeasured confounding in offline reinforcement learning from pixel-based demonstrations, where sensory mismatches cause biases, and introduces a causal objective that optimizes worst-case performance, achieving a 120% success rate compared to state-of-the-art methods.
Expressive policies based on flow-matching have been successfully applied in reinforcement learning (RL) more recently due to their ability to model complex action distributions from offline data. These algorithms build on standard policy gradients, which assume that there is no unmeasured confounding in the data. However, this condition does not necessarily hold for pixel-based demonstrations when a mismatch exists between the demonstrator's and the learner's sensory capabilities, leading to implicit confounding biases in offline data. We address the challenge by investigating the problem of confounded observations in offline RL from a causal perspective. We develop a novel causal offline RL objective that optimizes policies' worst-case performance that may arise due to confounding biases. Based on this new objective, we introduce a practical implementation that learns expressive flow-matching policies from confounded demonstrations, employing a deep discriminator to assess the discrepancy between the target policy and the nominal behavioral policy. Experiments across 25 pixel-based tasks demonstrate that our proposed confounding-robust augmentation procedure achieves a success rate 120\% that of confounding-unaware, state-of-the-art offline RL methods.