Data augmentation for efficient learning from parametric experts
This addresses the challenge of reducing environment interactions for policy cloning in reinforcement and imitation learning, with applications in expert compression and transfer from privileged experts, though it appears incremental as it builds on existing policy cloning methods.
The paper tackles the problem of data-efficient learning from parametric experts in reinforcement and imitation learning by introducing augmented policy cloning (APC), a data-augmentation technique that uses synthetic states to reduce environment interactions, achieving highly data-efficient transfer for high-degrees-of-freedom control problems.
We present a simple, yet powerful data-augmentation technique to enable data-efficient learning from parametric experts for reinforcement and imitation learning. We focus on what we call the policy cloning setting, in which we use online or offline queries of an expert or expert policy to inform the behavior of a student policy. This setting arises naturally in a number of problems, for instance as variants of behavior cloning, or as a component of other algorithms such as DAGGER, policy distillation or KL-regularized RL. Our approach, augmented policy cloning (APC), uses synthetic states to induce feedback-sensitivity in a region around sampled trajectories, thus dramatically reducing the environment interactions required for successful cloning of the expert. We achieve highly data-efficient transfer of behavior from an expert to a student policy for high-degrees-of-freedom control problems. We demonstrate the benefit of our method in the context of several existing and widely used algorithms that include policy cloning as a constituent part. Moreover, we highlight the benefits of our approach in two practically relevant settings (a) expert compression, i.e. transfer to a student with fewer parameters; and (b) transfer from privileged experts, i.e. where the expert has a different observation space than the student, usually including access to privileged information.