Periodic Intra-Ensemble Knowledge Distillation for Reinforcement Learning
This work addresses sample efficiency for reinforcement learning practitioners, but it is incremental as it builds on existing ensemble and knowledge distillation methods.
The paper tackles the problem of improving sample efficiency in reinforcement learning by proposing Periodic Intra-Ensemble Knowledge Distillation (PIEKD), which uses an ensemble of policies and periodically shares knowledge among them via distillation, resulting in improved performance on MuJoCo benchmark tasks.
Off-policy ensemble reinforcement learning (RL) methods have demonstrated impressive results across a range of RL benchmark tasks. Recent works suggest that directly imitating experts' policies in a supervised manner before or during the course of training enables faster policy improvement for an RL agent. Motivated by these recent insights, we propose Periodic Intra-Ensemble Knowledge Distillation (PIEKD). PIEKD is a learning framework that uses an ensemble of policies to act in the environment while periodically sharing knowledge amongst policies in the ensemble through knowledge distillation. Our experiments demonstrate that PIEKD improves upon a state-of-the-art RL method in sample efficiency on several challenging MuJoCo benchmark tasks. Additionally, we perform ablation studies to better understand PIEKD.