Beyond Conservatism: Diffusion Policies in Offline Multi-agent Reinforcement Learning
This addresses the challenge of data efficiency and robustness in multi-agent systems, offering a novel approach beyond conservative methods, though it appears incremental in applying diffusion models to a specific domain.
The paper tackles the problem of offline multi-agent reinforcement learning by introducing a diffusion-based model (DOM2) that enhances policy expressiveness and diversity, achieving state-of-the-art performance with over 20 times less data and better generalization in shifted environments.
We present a novel Diffusion Offline Multi-agent Model (DOM2) for offline Multi-Agent Reinforcement Learning (MARL). Different from existing algorithms that rely mainly on conservatism in policy design, DOM2 enhances policy expressiveness and diversity based on diffusion. Specifically, we incorporate a diffusion model into the policy network and propose a trajectory-based data-augmentation scheme in training. These key ingredients make our algorithm more robust to environment changes and achieve significant improvements in performance, generalization and data-efficiency. Our extensive experimental results demonstrate that DOM2 outperforms existing state-of-the-art methods in multi-agent particle and multi-agent MuJoCo environments, and generalizes significantly better in shifted environments thanks to its high expressiveness and diversity. Furthermore, DOM2 shows superior data efficiency and can achieve state-of-the-art performance with $20+$ times less data compared to existing algorithms.