Experience Augmentation: Boosting and Accelerating Off-Policy Multi-Agent Reinforcement Learning
This addresses the problem of slow and inefficient exploration in multi-agent RL for researchers and practitioners, offering an incremental improvement through a novel augmentation technique.
The paper tackles the challenge of exploring high-dimensional state-action spaces in multi-agent reinforcement learning by introducing Experience Augmentation, a technique that accelerates and boosts learning; in tests with MADDPG, it achieved convergence rewards in 1/4 the time and with better performance than the vanilla method.
Exploration of the high-dimensional state action space is one of the biggest challenges in Reinforcement Learning (RL), especially in multi-agent domain. We present a novel technique called Experience Augmentation, which enables a time-efficient and boosted learning based on a fast, fair and thorough exploration to the environment. It can be combined with arbitrary off-policy MARL algorithms and is applicable to either homogeneous or heterogeneous environments. We demonstrate our approach by combining it with MADDPG and verifing the performance in two homogeneous and one heterogeneous environments. In the best performing scenario, the MADDPG with experience augmentation reaches to the convergence reward of vanilla MADDPG with 1/4 realistic time, and its convergence beats the original model by a significant margin. Our ablation studies show that experience augmentation is a crucial ingredient which accelerates the training process and boosts the convergence.