Curriculum-based Asymmetric Multi-task Reinforcement Learning
This addresses the problem of efficient multi-task learning in reinforcement learning for AI researchers, though it appears incremental as it builds on existing curriculum and asymmetric multi-task learning approaches.
The paper tackles the challenge of negative transfer and training order in multi-task reinforcement learning by introducing CAMRL, a curriculum-based asymmetric multi-task learning algorithm that dynamically switches between training modes and uses a composite loss with ranking functions, achieving improvements over single-task and state-of-the-art methods across benchmarks like Gym-minigrid, Meta-world, Atari, PyBullet, and RLBench.
We introduce CAMRL, the first curriculum-based asymmetric multi-task learning (AMTL) algorithm for dealing with multiple reinforcement learning (RL) tasks altogether. To mitigate the negative influence of customizing the one-off training order in curriculum-based AMTL, CAMRL switches its training mode between parallel single-task RL and asymmetric multi-task RL (MTRL), according to an indicator regarding the training time, the overall performance, and the performance gap among tasks. To leverage the multi-sourced prior knowledge flexibly and to reduce negative transfer in AMTL, we customize a composite loss with multiple differentiable ranking functions and optimize the loss through alternating optimization and the Frank-Wolfe algorithm. The uncertainty-based automatic adjustment of hyper-parameters is also applied to eliminate the need of laborious hyper-parameter analysis during optimization. By optimizing the composite loss, CAMRL predicts the next training task and continuously revisits the transfer matrix and network weights. We have conducted experiments on a wide range of benchmarks in multi-task RL, covering Gym-minigrid, Meta-world, Atari video games, vision-based PyBullet tasks, and RLBench, to show the improvements of CAMRL over the corresponding single-task RL algorithm and state-of-the-art MTRL algorithms. The code is available at: https://github.com/huanghanchi/CAMRL