Transfer learning with causal counterfactual reasoning in Decision Transformers
This addresses the challenge of adaptation in reinforcement learning for improved flexibility and efficiency, though it appears incremental as it builds on existing Decision Transformer methods.
The paper tackles the problem of transferring learned policies to new environments with unseen dynamics by using causal counterfactual reasoning in Decision Transformers, resulting in a bootstrapped policy that retains most of the reward.
The ability to adapt to changes in environmental contingencies is an important challenge in reinforcement learning. Indeed, transferring previously acquired knowledge to environments with unseen structural properties can greatly enhance the flexibility and efficiency by which novel optimal policies may be constructed. In this work, we study the problem of transfer learning under changes in the environment dynamics. In this study, we apply causal reasoning in the offline reinforcement learning setting to transfer a learned policy to new environments. Specifically, we use the Decision Transformer (DT) architecture to distill a new policy on the new environment. The DT is trained on data collected by performing policy rollouts on factual and counterfactual simulations from the source environment. We show that this mechanism can bootstrap a successful policy on the target environment while retaining most of the reward.