Noisy Zero-Shot Coordination: Breaking The Common Knowledge Assumption In Zero-Shot Coordination Games
This work addresses the challenge of coordination in multi-agent reinforcement learning for scenarios where environment specifications are imperfect, representing an incremental improvement over prior ZSC methods by relaxing a key assumption.
The paper tackles the problem of zero-shot coordination (ZSC) by addressing its reliance on the common knowledge assumption, which is often invalid in real-world settings, and introduces noisy zero-shot coordination (NZSC) where agents observe noisy versions of the environment. The result shows that with NZSC training, RL agents can coordinate well with novel partners even without exact common knowledge, as demonstrated through a reduction to a meta-Dec-POMDP and a proposed meta-learning method.
Zero-shot coordination (ZSC) is a popular setting for studying the ability of reinforcement learning (RL) agents to coordinate with novel partners. Prior ZSC formulations assume the $\textit{problem setting}$ is common knowledge: each agent knows the underlying Dec-POMDP, knows others have this knowledge, and so on ad infinitum. However, this assumption rarely holds in complex real-world settings, which are often difficult to fully and correctly specify. Hence, in settings where this common knowledge assumption is invalid, agents trained using ZSC methods may not be able to coordinate well. To address this limitation, we formulate the $\textit{noisy zero-shot coordination}$ (NZSC) problem. In NZSC, agents observe different noisy versions of the ground truth Dec-POMDP, which are assumed to be distributed according to a fixed noise model. Only the distribution of ground truth Dec-POMDPs and the noise model are common knowledge. We show that a NZSC problem can be reduced to a ZSC problem by designing a meta-Dec-POMDP with an augmented state space consisting of all the ground-truth Dec-POMDPs. For solving NZSC problems, we propose a simple and flexible meta-learning method called NZSC training, in which the agents are trained across a distribution of coordination problems - which they only get to observe noisy versions of. We show that with NZSC training, RL agents can be trained to coordinate well with novel partners even when the (exact) problem setting of the coordination is not common knowledge.