Stability of Multi-Agent Learning in Competitive Networks: Delaying the Onset of Chaos
This addresses the problem of predicting and controlling chaotic behaviors in multi-agent learning for AI and game theory researchers, though it is incremental as it builds on prior work by lifting the zero-sum assumption.
The study investigates Q-Learning dynamics in competitive network games beyond zero-sum assumptions, finding that stability depends on network connectivity rather than the total number of agents, with experiments showing increased agent counts can avoid instability under certain network structures.
The behaviour of multi-agent learning in competitive network games is often studied within the context of zero-sum games, in which convergence guarantees may be obtained. However, outside of this class the behaviour of learning is known to display complex behaviours and convergence cannot be always guaranteed. Nonetheless, in order to develop a complete picture of the behaviour of multi-agent learning in competitive settings, the zero-sum assumption must be lifted. Motivated by this we study the Q-Learning dynamics, a popular model of exploration and exploitation in multi-agent learning, in competitive network games. We determine how the degree of competition, exploration rate and network connectivity impact the convergence of Q-Learning. To study generic competitive games, we parameterise network games in terms of correlations between agent payoffs and study the average behaviour of the Q-Learning dynamics across all games drawn from a choice of this parameter. This statistical approach establishes choices of parameters for which Q-Learning dynamics converge to a stable fixed point. Differently to previous works, we find that the stability of Q-Learning is explicitly dependent only on the network connectivity rather than the total number of agents. Our experiments validate these findings and show that, under certain network structures, the total number of agents can be increased without increasing the likelihood of unstable or chaotic behaviours.