Learning Symbolic Task Decompositions for Multi-Agent Teams
This addresses sample efficiency for multi-agent teams in reinforcement learning, though it is incremental as it builds on reward machines and task decomposition concepts.
The paper tackles the problem of improving sample efficiency in cooperative multi-agent learning by learning optimal task decompositions from model-free interactions, removing the need for manual design. It demonstrates efficacy in deep reinforcement learning settings, enabling synchronous multi-agent learning in environments with codependent agent dynamics.
One approach for improving sample efficiency in cooperative multi-agent learning is to decompose overall tasks into sub-tasks that can be assigned to individual agents. We study this problem in the context of reward machines: symbolic tasks that can be formally decomposed into sub-tasks. In order to handle settings without a priori knowledge of the environment, we introduce a framework that can learn the optimal decomposition from model-free interactions with the environment. Our method uses a task-conditioned architecture to simultaneously learn an optimal decomposition and the corresponding agents' policies for each sub-task. In doing so, we remove the need for a human to manually design the optimal decomposition while maintaining the sample-efficiency benefits of improved credit assignment. We provide experimental results in several deep reinforcement learning settings, demonstrating the efficacy of our approach. Our results indicate that our approach succeeds even in environments with codependent agent dynamics, enabling synchronous multi-agent learning not achievable in previous works.