Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning
This work addresses scalable goal-reaching in AI, particularly for robotics, by enabling multistep stitching from offline visual datasets, though it appears incremental as it builds on existing GCRL and quasimetric methods.
The paper tackled the challenge of estimating temporal distances for long-horizon goal-conditioned reinforcement learning by integrating multistep Monte-Carlo returns into a quasimetric learning method, achieving superior performance on simulated tasks with up to 4000 steps and enabling real-world robotic manipulation from unlabeled visual data.
Learning how to reach goals in an environment is a longstanding challenge in AI, yet reasoning over long horizons remains a challenge for modern methods. The key question is how to estimate the temporal distance between pairs of observations. While temporal difference methods leverage local updates to provide optimality guarantees, they often perform worse than Monte Carlo methods that perform global updates (e.g., with multi-step returns), which lack such guarantees. We show how these approaches can be integrated into a practical GCRL method that fits a quasimetric distance using a multistep Monte-Carlo return. We show our method outperforms existing GCRL methods on long-horizon simulated tasks with up to 4000 steps, even with visual observations. We also demonstrate that our method can enable stitching in the real-world robotic manipulation domain (Bridge setup). Our approach is the first end-to-end GCRL method that enables multistep stitching in this real-world manipulation domain from an unlabeled offline dataset of visual observations.