Real World Offline Reinforcement Learning with Realistic Data Source
This work addresses the gap in ORL benchmarks for real-world robotics, providing insights for robot learning practitioners, though it is incremental in applying existing methods to new data.
The authors tackled the problem of offline reinforcement learning (ORL) in real-world robotics by evaluating ORL methods on practical data sources from safe operations of related tasks, finding that ORL can generalize from heterogeneous data and outperform imitation learning in tabletop manipulation tasks.
Offline reinforcement learning (ORL) holds great promise for robot learning due to its ability to learn from arbitrary pre-generated experience. However, current ORL benchmarks are almost entirely in simulation and utilize contrived datasets like replay buffers of online RL agents or sub-optimal trajectories, and thus hold limited relevance for real-world robotics. In this work (Real-ORL), we posit that data collected from safe operations of closely related tasks are more practical data sources for real-world robot learning. Under these settings, we perform an extensive (6500+ trajectories collected over 800+ robot hours and 270+ human labor hour) empirical study evaluating generalization and transfer capabilities of representative ORL methods on four real-world tabletop manipulation tasks. Our study finds that ORL and imitation learning prefer different action spaces, and that ORL algorithms can generalize from leveraging offline heterogeneous data sources and outperform imitation learning. We release our dataset and implementations at URL: https://sites.google.com/view/real-orl