Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairs
This addresses the problem of inefficient training in reinforcement learning for AI agents by enabling automatic generation of solvable task-level pairs, representing an incremental advance over prior unsupervised environment design methods that only considered fixed tasks.
The paper tackles the challenge of training general agents to follow complex instructions in intricate environments by co-designing tasks and levels, introducing ATLAS which generates joint autocurricula over tasks and levels and outperforms random sampling approaches, especially when solvable pairs are unlikely.
Training general agents to follow complex instructions (tasks) in intricate environments (levels) remains a core challenge in reinforcement learning. Random sampling of task-level pairs often produces unsolvable combinations, highlighting the need to co-design tasks and levels. While unsupervised environment design (UED) has proven effective at automatically designing level curricula, prior work has only considered a fixed task. We present ATLAS (Aligning Tasks and Levels for Autocurricula of Specifications), a novel method that generates joint autocurricula over tasks and levels. Our approach builds upon UED to automatically produce solvable yet challenging task-level pairs for policy training. To evaluate ATLAS and drive progress in the field, we introduce an evaluation suite that models tasks as reward machines in Minigrid levels. Experiments demonstrate that ATLAS vastly outperforms random sampling approaches, particularly when sampling solvable pairs is unlikely. We further show that mutations leveraging the structure of both tasks and levels accelerate convergence to performant policies.