Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning
This work addresses the challenge of multi-task learning in multi-agent reinforcement learning for applications requiring complex, non-Markovian specifications, representing an incremental extension from single-agent to multi-agent contexts.
The authors tackled the problem of enabling multiple reinforcement learning agents to concurrently learn various non-Markovian specifications by extending formal methods to multi-agent settings, resulting in the formal definition of Extended Markov Games and empirical tests showing successful training of logic-based multi-agent RL algorithms on diverse non-Markovian co-safe LTL specifications.
The combination of Formal Methods with Reinforcement Learning (RL) has recently attracted interest as a way for single-agent RL to learn multiple-task specifications. In this paper we extend this convergence to multi-agent settings and formally define Extended Markov Games as a general mathematical model that allows multiple RL agents to concurrently learn various non-Markovian specifications. To introduce this new model we provide formal definitions and proofs as well as empirical tests of RL algorithms running on this framework. Specifically, we use our model to train two different logic-based multi-agent RL algorithms to solve diverse settings of non-Markovian co-safe LTL specifications.