Logical Team Q-learning: An approach towards factored policies in cooperative MARL
This work addresses the challenge of optimizing individual agent behaviors for optimal joint policies in cooperative MARL, which is incremental as it builds on existing MARL methods.
The paper tackles the problem of learning factored policies in cooperative multi-agent reinforcement learning (MARL) by introducing Logical Team Q-learning (LTQL), a method that does not rely on environment assumptions and is applicable to any collaborative scenario, with experiments in tabular and deep settings illustrating its effectiveness.
We address the challenge of learning factored policies in cooperative MARL scenarios. In particular, we consider the situation in which a team of agents collaborates to optimize a common cost. The goal is to obtain factored policies that determine the individual behavior of each agent so that the resulting joint policy is optimal. The main contribution of this work is the introduction of Logical Team Q-learning (LTQL). LTQL does not rely on assumptions about the environment and hence is generally applicable to any collaborative MARL scenario. We derive LTQL as a stochastic approximation to a dynamic programming method we introduce in this work. We conclude the paper by providing experiments (both in the tabular and deep settings) that illustrate the claims.