High-order Interactions Modeling for Interpretable Multi-Agent Q-Learning
This work addresses the challenge of interpretable coordination in multi-agent systems, offering a solution that is incremental but provides specific gains in performance and transparency.
The paper tackles the problem of modeling high-order interactions in multi-agent reinforcement learning, which is hindered by combinatorial explosion and opacity, by proposing a novel value decomposition framework called Continued Fraction Q-Learning (QCoFr) that captures arbitrary-order interactions with linear complexity and enhances cooperation and interpretability through a variational information bottleneck.
The ability to model interactions among agents is crucial for effective coordination and understanding their cooperation mechanisms in multi-agent reinforcement learning (MARL). However, previous efforts to model high-order interactions have been primarily hindered by the combinatorial explosion or the opaque nature of their black-box network structures. In this paper, we propose a novel value decomposition framework, called Continued Fraction Q-Learning (QCoFr), which can flexibly capture arbitrary-order agent interactions with only linear complexity $\mathcal{O}\left({n}\right)$ in the number of agents, thus avoiding the combinatorial explosion when modeling rich cooperation. Furthermore, we introduce the variational information bottleneck to extract latent information for estimating credits. This latent information helps agents filter out noisy interactions, thereby significantly enhancing both cooperation and interpretability. Extensive experiments demonstrate that QCoFr not only consistently achieves better performance but also provides interpretability that aligns with our theoretical analysis.