Actor-Critic Pretraining for Proximal Policy Optimization
This addresses sample inefficiency in robotics applications, offering an incremental improvement over existing pretraining methods.
The paper tackles the problem of high sample inefficiency in reinforcement learning for robotics by proposing actor-critic pretraining using expert demonstrations, resulting in an average 86.1% improvement in sample efficiency compared to no pretraining and 30.9% over actor-only pretraining.
Reinforcement learning (RL) actor-critic algorithms enable autonomous learning but often require a large number of environment interactions, which limits their applicability in robotics. Leveraging expert data can reduce the number of required environment interactions. A common approach is actor pretraining, where the actor network is initialized via behavioral cloning on expert demonstrations and subsequently fine-tuned with RL. In contrast, the initialization of the critic network has received little attention, despite its central role in policy optimization. This paper proposes a pretraining approach for actor-critic algorithms like Proximal Policy Optimization (PPO) that uses expert demonstrations to initialize both networks. The actor is pretrained via behavioral cloning, while the critic is pretrained using returns obtained from rollouts of the pretrained policy. The approach is evaluated on 15 simulated robotic manipulation and locomotion tasks. Experimental results show that actor-critic pretraining improves sample efficiency by 86.1% on average compared to no pretraining and by 30.9% to actor-only pretraining.