Online Robust Multi-Agent Reinforcement Learning under Model Uncertainties
This work addresses the challenge of developing robust multi-agent systems under model uncertainties for applications like robotics or autonomous vehicles, representing a novel approach rather than an incremental improvement.
The paper tackles the problem of multi-agent systems failing due to model mismatches in real-world environments by introducing online learning for Distributionally Robust Markov Games, where agents learn directly from interactions without prior data. It proposes the Robust Optimistic Nash Value Iteration algorithm, achieving low regret and optimal robust policies with provable guarantees for uncertainty sets measured by Total Variation and Kullback-Leibler divergences.
Well-trained multi-agent systems can fail when deployed in real-world environments due to model mismatches between the training and deployment environments, caused by environment uncertainties including noise or adversarial attacks. Distributionally Robust Markov Games (DRMGs) enhance system resilience by optimizing for worst-case performance over a defined set of environmental uncertainties. However, current methods are limited by their dependence on simulators or large offline datasets, which are often unavailable. This paper pioneers the study of online learning in DRMGs, where agents learn directly from environmental interactions without prior data. We introduce the {\it Robust Optimistic Nash Value Iteration (RONAVI)} algorithm and provide the first provable guarantees for this setting. Our theoretical analysis demonstrates that the algorithm achieves low regret and efficiently finds the optimal robust policy for uncertainty sets measured by Total Variation divergence and Kullback-Leibler divergence. These results establish a new, practical path toward developing truly robust multi-agent systems.