Reinforcement Learning with Success Induced Task Prioritization
This addresses the challenge of automating curriculum design for RL practitioners, though it is incremental as it builds on existing curriculum learning frameworks.
The paper tackles the problem of designing effective task curricula for reinforcement learning by introducing Success Induced Task Prioritization (SITP), which sequences tasks based on success rates to accelerate learning, and demonstrates that it matches or surpasses other curriculum methods in benchmarks like POGEMA and Procgen.
Many challenging reinforcement learning (RL) problems require designing a distribution of tasks that can be applied to train effective policies. This distribution of tasks can be specified by the curriculum. A curriculum is meant to improve the results of learning and accelerate it. We introduce Success Induced Task Prioritization (SITP), a framework for automatic curriculum learning, where a task sequence is created based on the success rate of each task. In this setting, each task is an algorithmically created environment instance with a unique configuration. The algorithm selects the order of tasks that provide the fastest learning for agents. The probability of selecting any of the tasks for the next stage of learning is determined by evaluating its performance score in previous stages. Experiments were carried out in the Partially Observable Grid Environment for Multiple Agents (POGEMA) and Procgen benchmark. We demonstrate that SITP matches or surpasses the results of other curriculum design methods. Our method can be implemented with handful of minor modifications to any standard RL framework and provides useful prioritization with minimal computational overhead.