Offline Decentralized Multi-Agent Reinforcement Learning
This addresses the challenge of learning coordinated policies in multi-agent systems when online interaction is costly or risky, representing an incremental advance in offline RL methods.
The paper tackles the problem of offline decentralized multi-agent reinforcement learning, where agents learn from static datasets without environment interaction, by proposing a framework that uses value deviation and transition normalization to modify transition probabilities, resulting in agents learning high-performing and coordinated policies with substantial empirical improvements in various tasks.
In many real-world multi-agent cooperative tasks, due to high cost and risk, agents cannot continuously interact with the environment and collect experiences during learning, but have to learn from offline datasets. However, the transition dynamics in the dataset of each agent can be much different from the ones induced by the learned policies of other agents in execution, creating large errors in value estimates. Consequently, agents learn uncoordinated low-performing policies. In this paper, we propose a framework for offline decentralized multi-agent reinforcement learning, which exploits value deviation and transition normalization to deliberately modify the transition probabilities. Value deviation optimistically increases the transition probabilities of high-value next states, and transition normalization normalizes the transition probabilities of next states. They together enable agents to learn high-performing and coordinated policies. Theoretically, we prove the convergence of Q-learning under the altered non-stationary transition dynamics. Empirically, we show that the framework can be easily built on many existing offline reinforcement learning algorithms and achieve substantial improvement in a variety of multi-agent tasks.