Macro-Action-Based Deep Multi-Agent Reinforcement Learning
This work addresses the challenge of enabling robots to collaborate effectively in real-world tasks by introducing scalable learning methods for asynchronous macro-actions, representing an incremental advancement in multi-agent reinforcement learning.
The paper tackled the problem of asynchronous decision-making in multi-robot systems by developing deep reinforcement learning methods for macro-action-based frameworks, resulting in improved performance and scalability over primitive-action approaches in benchmark evaluations.
In real-world multi-robot systems, performing high-quality, collaborative behaviors requires robots to asynchronously reason about high-level action selection at varying time durations. Macro-Action Decentralized Partially Observable Markov Decision Processes (MacDec-POMDPs) provide a general framework for asynchronous decision making under uncertainty in fully cooperative multi-agent tasks. However, multi-agent deep reinforcement learning methods have only been developed for (synchronous) primitive-action problems. This paper proposes two Deep Q-Network (DQN) based methods for learning decentralized and centralized macro-action-value functions with novel macro-action trajectory replay buffers introduced for each case. Evaluations on benchmark problems and a larger domain demonstrate the advantage of learning with macro-actions over primitive-actions and the scalability of our approaches.