Optimal Transport for Offline Imitation Learning
This addresses the challenge of reward engineering in offline RL for practitioners, though it is incremental as it builds on existing offline RL methods.
The paper tackles the problem of offline reinforcement learning requiring reward-annotated datasets by introducing Optimal Transport Reward labeling (OTR), which assigns rewards using optimal transport alignment with a few demonstrations, and shows that OTR with a single demonstration matches the performance of offline RL with ground-truth rewards on D4RL benchmarks.
With the advent of large datasets, offline reinforcement learning (RL) is a promising framework for learning good decision-making policies without the need to interact with the real environment. However, offline RL requires the dataset to be reward-annotated, which presents practical challenges when reward engineering is difficult or when obtaining reward annotations is labor-intensive. In this paper, we introduce Optimal Transport Reward labeling (OTR), an algorithm that assigns rewards to offline trajectories, with a few high-quality demonstrations. OTR's key idea is to use optimal transport to compute an optimal alignment between an unlabeled trajectory in the dataset and an expert demonstration to obtain a similarity measure that can be interpreted as a reward, which can then be used by an offline RL algorithm to learn the policy. OTR is easy to implement and computationally efficient. On D4RL benchmarks, we show that OTR with a single demonstration can consistently match the performance of offline RL with ground-truth rewards.