Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformer
This addresses a bottleneck in reinforcement learning for real-world applications with limited or suboptimal data, offering an incremental improvement in generalization.
The paper tackles the problem of Decision Transformers requiring high-quality offline data by proposing the Counterfactual Reasoning Decision Transformer (CRDT), which generates counterfactual experiences to improve decision-making, resulting in outperformance over conventional methods on Atari and D4RL benchmarks.
Decision Transformers (DT) play a crucial role in modern reinforcement learning, leveraging offline datasets to achieve impressive results across various domains. However, DT requires high-quality, comprehensive data to perform optimally. In real-world applications, the lack of training data and the scarcity of optimal behaviours make training on offline datasets challenging, as suboptimal data can hinder performance. To address this, we propose the Counterfactual Reasoning Decision Transformer (CRDT), a novel framework inspired by counterfactual reasoning. CRDT enhances DT ability to reason beyond known data by generating and utilizing counterfactual experiences, enabling improved decision-making in unseen scenarios. Experiments across Atari and D4RL benchmarks, including scenarios with limited data and altered dynamics, demonstrate that CRDT outperforms conventional DT approaches. Additionally, reasoning counterfactually allows the DT agent to obtain stitching abilities, combining suboptimal trajectories, without architectural modifications. These results highlight the potential of counterfactual reasoning to enhance reinforcement learning agents' performance and generalization capabilities.