Variational Policy Propagation for Multi-agent Reinforcement Learning
This addresses the challenge of scalable and effective collaboration in multi-agent systems, though it appears incremental as it builds on existing methods with specific improvements.
The paper tackles the problem of learning joint policies in multi-agent reinforcement learning by proposing Variational Policy Propagation (VPP), which models the joint policy as a Markov Random Field and integrates variational inference for efficient sampling, resulting in outperforming previous state-of-the-art methods on large-scale tasks.
We propose a \emph{collaborative} multi-agent reinforcement learning algorithm named variational policy propagation (VPP) to learn a \emph{joint} policy through the interactions over agents. We prove that the joint policy is a Markov Random Field under some mild conditions, which in turn reduces the policy space effectively. We integrate the variational inference as special differentiable layers in policy such that the actions can be efficiently sampled from the Markov Random Field and the overall policy is differentiable. We evaluate our algorithm on several large scale challenging tasks and demonstrate that it outperforms previous state-of-the-arts.