Real-time Policy Distillation in Deep Reinforcement Learning
This work addresses the problem of slow and limited policy distillation for researchers and practitioners in deep reinforcement learning, offering an incremental improvement over existing methods.
The paper tackles the computational inefficiency and model capacity limitations of policy distillation in deep reinforcement learning by proposing a real-time distillation mechanism where teacher training and student distillation occur simultaneously, reducing distillation time by at least half and enabling expert-level performance in extremely small student models with compression ratios as low as 1.7% in Atari 2600 games.
Policy distillation in deep reinforcement learning provides an effective way to transfer control policies from a larger network to a smaller untrained network without a significant degradation in performance. However, policy distillation is underexplored in deep reinforcement learning, and existing approaches are computationally inefficient, resulting in a long distillation time. In addition, the effectiveness of the distillation process is still limited to the model capacity. We propose a new distillation mechanism, called real-time policy distillation, in which training the teacher model and distilling the policy to the student model occur simultaneously. Accordingly, the teacher's latest policy is transferred to the student model in real time. This reduces the distillation time to half the original time or even less and also makes it possible for extremely small student models to learn skills at the expert level. We evaluated the proposed algorithm in the Atari 2600 domain. The results show that our approach can achieve full distillation in most games, even with compression ratios up to 1.7%.