Distance Weighted Supervised Learning for Offline Interaction Data
This addresses the problem of leveraging diverse offline data for sequential decision-making in robotics or AI, offering a robust supervised approach without bootstrapping, though it appears incremental in bridging imitation learning and reinforcement learning.
The paper tackles the challenge of learning goal-conditioned policies from sub-optimal offline interaction data by introducing Distance Weighted Supervised Learning (DWSL), which models time-step distributions to approximate shortest path distances and weights actions accordingly, achieving performance improvements over prior methods in high-dimensional image domains.
Sequential decision making algorithms often struggle to leverage different sources of unstructured offline interaction data. Imitation learning (IL) methods based on supervised learning are robust, but require optimal demonstrations, which are hard to collect. Offline goal-conditioned reinforcement learning (RL) algorithms promise to learn from sub-optimal data, but face optimization challenges especially with high-dimensional data. To bridge the gap between IL and RL, we introduce Distance Weighted Supervised Learning or DWSL, a supervised method for learning goal-conditioned policies from offline data. DWSL models the entire distribution of time-steps between states in offline data with only supervised learning, and uses this distribution to approximate shortest path distances. To extract a policy, we weight actions by their reduction in distance estimates. Theoretically, DWSL converges to an optimal policy constrained to the data distribution, an attractive property for offline learning, without any bootstrapping. Across all datasets we test, DWSL empirically maintains behavior cloning as a lower bound while still exhibiting policy improvement. In high-dimensional image domains, DWSL surpasses the performance of both prior goal-conditioned IL and RL algorithms. Visualizations and code can be found at https://sites.google.com/view/dwsl/home .