Learning Nash Equilibria in Zero-Sum Stochastic Games via Entropy-Regularized Policy Approximation
This addresses efficiency in multi-agent reinforcement learning for game theory applications, though it appears incremental as an enhancement to existing Q-learning methods.
The paper tackles the computational cost of learning Nash equilibria in zero-sum stochastic games by proposing a Q-learning algorithm with entropy-regularized soft policies, achieving a major speed-up over existing algorithms in empirical tests.
We explore the use of policy approximations to reduce the computational cost of learning Nash equilibria in zero-sum stochastic games. We propose a new Q-learning type algorithm that uses a sequence of entropy-regularized soft policies to approximate the Nash policy during the Q-function updates. We prove that under certain conditions, by updating the regularized Q-function, the algorithm converges to a Nash equilibrium. We also demonstrate the proposed algorithm's ability to transfer previous training experiences, enabling the agents to adapt quickly to new environments. We provide a dynamic hyper-parameter scheduling scheme to further expedite convergence. Empirical results applied to a number of stochastic games verify that the proposed algorithm converges to the Nash equilibrium, while exhibiting a major speed-up over existing algorithms.