Zero-Shot Off-Policy Learning
This work addresses the challenge of distributional shift and overestimation in off-policy learning for zero-shot adaptation, benefiting researchers in reinforcement learning and robotics.
The paper tackles the problem of off-policy learning in zero-shot reinforcement learning, where an agent must adapt to new tasks without training, by discovering a theoretical connection between successor measures and stationary density ratios, enabling optimal policy inference and achieving competitive performance in benchmarks like SMPL Humanoid and ExoRL.
Off-policy learning methods seek to derive an optimal policy directly from a fixed dataset of prior interactions. This objective presents significant challenges, primarily due to the inherent distributional shift and value function overestimation bias. These issues become even more noticeable in zero-shot reinforcement learning, where an agent trained on reward-free data must adapt to new tasks at test time without additional training. In this work, we address the off-policy problem in a zero-shot setting by discovering a theoretical connection of successor measures to stationary density ratios. Using this insight, our algorithm can infer optimal importance sampling ratios, effectively performing a stationary distribution correction with an optimal policy for any task on the fly. We benchmark our method in motion tracking tasks on SMPL Humanoid, continuous control on ExoRL, and for the long-horizon OGBench tasks. Our technique seamlessly integrates into forward-backward representation frameworks and enables fast-adaptation to new tasks in a training-free regime. More broadly, this work bridges off-policy learning and zero-shot adaptation, offering benefits to both research areas.