Debiased Offline Representation Learning for Fast Online Adaptation in Non-stationary Dynamics
This work addresses the challenge of fast online adaptation in non-stationary dynamics for real-world reinforcement learning applications, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackled the problem of learning adaptable policies in offline reinforcement learning for non-stationary environments by addressing context misassociations due to limited data, resulting in DORA achieving more precise dynamics encoding and significantly outperforming baselines across six benchmark tasks.
Developing policies that can adjust to non-stationary environments is essential for real-world reinforcement learning applications. However, learning such adaptable policies in offline settings, with only a limited set of pre-collected trajectories, presents significant challenges. A key difficulty arises because the limited offline data makes it hard for the context encoder to differentiate between changes in the environment dynamics and shifts in the behavior policy, often leading to context misassociations. To address this issue, we introduce a novel approach called Debiased Offline Representation for fast online Adaptation (DORA). DORA incorporates an information bottleneck principle that maximizes mutual information between the dynamics encoding and the environmental data, while minimizing mutual information between the dynamics encoding and the actions of the behavior policy. We present a practical implementation of DORA, leveraging tractable bounds of the information bottleneck principle. Our experimental evaluation across six benchmark MuJoCo tasks with variable parameters demonstrates that DORA not only achieves a more precise dynamics encoding but also significantly outperforms existing baselines in terms of performance.