Distillation Policy Optimization
This addresses a key challenge in reinforcement learning for researchers and practitioners by offering a more stable and efficient learning method, though it appears incremental as it builds on existing actor-critic approaches.
The paper tackles the problem of balancing sample efficiency and stability in reinforcement learning by introducing an actor-critic framework that integrates on- and off-policy data, resulting in substantial improvements in sample efficiency for on-policy algorithms.
While on-policy algorithms are known for their stability, they often demand a substantial number of samples. In contrast, off-policy algorithms, which leverage past experiences, are considered sample-efficient but tend to exhibit instability. Can we develop an algorithm that harnesses the benefits of off-policy data while maintaining stable learning? In this paper, we introduce an actor-critic learning framework that harmonizes two data sources for both evaluation and control, facilitating rapid learning and adaptable integration with on-policy algorithms. This framework incorporates variance reduction mechanisms, including a unified advantage estimator (UAE) and a residual baseline, improving the efficacy of both on- and off-policy learning. Our empirical results showcase substantial enhancements in sample efficiency for on-policy algorithms, effectively bridging the gap to the off-policy approaches. It demonstrates the promise of our approach as a novel learning paradigm.