Chaos, Extremism and Optimism: Volume Analysis of Learning in Games
This provides theoretical insights into learning dynamics in games, addressing instability and convergence issues for researchers in game theory and machine learning, though it is incremental as it builds on existing methods.
The paper analyzes Multiplicative Weights Updates (MWU) and Optimistic MWU (OMWU) in zero-sum and coordination games using volume analysis in dual space, revealing that MWU expands volume (chaotic) in zero-sum games while OMWU contracts it (convergent), but roles reverse in coordination games. It proves negative properties of MWU in zero-sum games: extremism (recurrent sticking near pure strategies) and unavoidability (cannot avoid bad points).
We present volume analyses of Multiplicative Weights Updates (MWU) and Optimistic Multiplicative Weights Updates (OMWU) in zero-sum as well as coordination games. Such analyses provide new insights into these game dynamical systems, which seem hard to achieve via the classical techniques within Computer Science and Machine Learning. The first step is to examine these dynamics not in their original space (simplex of actions) but in a dual space (aggregate payoff space of actions). The second step is to explore how the volume of a set of initial conditions evolves over time when it is pushed forward according to the algorithm. This is reminiscent of approaches in Evolutionary Game Theory where replicator dynamics, the continuous-time analogue of MWU, is known to always preserve volume in all games. Interestingly, when we examine discrete-time dynamics, both the choice of the game and the choice of the algorithm play a critical role. So whereas MWU expands volume in zero-sum games and is thus Lyapunov chaotic, we show that OMWU contracts volume, providing an alternative understanding for its known convergent behavior. However, we also prove a no-free-lunch type of theorem, in the sense that when examining coordination games the roles are reversed: OMWU expands volume exponentially fast, whereas MWU contracts. Using these tools, we prove two novel, rather negative properties of MWU in zero-sum games: (1) Extremism: even in games with unique fully mixed Nash equilibrium, the system recurrently gets stuck near pure-strategy profiles, despite them being clearly unstable from game theoretic perspective. (2) Unavoidability: given any set of good points (with your own interpretation of "good"), the system cannot avoid bad points indefinitely.