Rethinking Optimal Transport in Offline Reinforcement Learning
This addresses the challenge of policy stitching in offline RL for robotics and control applications, though it appears incremental as it builds on existing optimal transport concepts.
The authors tackled the problem of extracting efficient policies from suboptimal expert data in offline reinforcement learning by reformulating it as an optimal transport problem, resulting in a novel algorithm that shows performance improvements on D4RL continuous control benchmarks.
We propose a novel algorithm for offline reinforcement learning using optimal transport. Typically, in offline reinforcement learning, the data is provided by various experts and some of them can be sub-optimal. To extract an efficient policy, it is necessary to \emph{stitch} the best behaviors from the dataset. To address this problem, we rethink offline reinforcement learning as an optimal transportation problem. And based on this, we present an algorithm that aims to find a policy that maps states to a \emph{partial} distribution of the best expert actions for each given state. We evaluate the performance of our algorithm on continuous control problems from the D4RL suite and demonstrate improvements over existing methods.