Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation
This work addresses the challenge of formalizing task distribution interpolation in CRL for improved training in robotics, though it is incremental as it builds on existing ideas from gradual domain adaptation.
The paper tackles the problem of generating effective task sequences in Curriculum Reinforcement Learning by framing it as an optimal transport problem, proposing GRADIENT to create curricula via Wasserstein barycenters, and shows it achieves higher learning efficiency and asymptotic performance in locomotion and manipulation tasks.
Curriculum Reinforcement Learning (CRL) aims to create a sequence of tasks, starting from easy ones and gradually learning towards difficult tasks. In this work, we focus on the idea of framing CRL as interpolations between a source (auxiliary) and a target task distribution. Although existing studies have shown the great potential of this idea, it remains unclear how to formally quantify and generate the movement between task distributions. Inspired by the insights from gradual domain adaptation in semi-supervised learning, we create a natural curriculum by breaking down the potentially large task distributional shift in CRL into smaller shifts. We propose GRADIENT, which formulates CRL as an optimal transport problem with a tailored distance metric between tasks. Specifically, we generate a sequence of task distributions as a geodesic interpolation (i.e., Wasserstein barycenter) between the source and target distributions. Different from many existing methods, our algorithm considers a task-dependent contextual distance metric and is capable of handling nonparametric distributions in both continuous and discrete context settings. In addition, we theoretically show that GRADIENT enables smooth transfer between subsequent stages in the curriculum under certain conditions. We conduct extensive experiments in locomotion and manipulation tasks and show that our proposed GRADIENT achieves higher performance than baselines in terms of learning efficiency and asymptotic performance.