Proximal Curriculum for Reinforcement Learning Agents
This addresses the challenge of efficient training for RL agents, though it appears incremental as it builds on existing curriculum design techniques.
The paper tackles the problem of curriculum design for reinforcement learning agents in multi-task settings by introducing ProCuRL, a strategy inspired by the Zone of Proximal Development, which accelerates training over state-of-the-art baselines.
We consider the problem of curriculum design for reinforcement learning (RL) agents in contextual multi-task settings. Existing techniques on automatic curriculum design typically require domain-specific hyperparameter tuning or have limited theoretical underpinnings. To tackle these limitations, we design our curriculum strategy, ProCuRL, inspired by the pedagogical concept of Zone of Proximal Development (ZPD). ProCuRL captures the intuition that learning progress is maximized when picking tasks that are neither too hard nor too easy for the learner. We mathematically derive ProCuRL by analyzing two simple learning settings. We also present a practical variant of ProCuRL that can be directly integrated with deep RL frameworks with minimal hyperparameter tuning. Experimental results on a variety of domains demonstrate the effectiveness of our curriculum strategy over state-of-the-art baselines in accelerating the training process of deep RL agents.