A Multi-Step Minimax Q-learning Algorithm for Two-Player Zero-Sum Markov Games
This work addresses a specific problem in game theory and reinforcement learning for researchers in those fields, presenting an incremental improvement with theoretical guarantees.
The authors tackled the problem of solving two-player zero-sum Markov games without known model information by proposing a multi-step minimax Q-learning algorithm, which theoretically converges to the game-theoretic optimal value with probability one and is shown to be effective and easy to implement in numerical simulations.
An interesting iterative procedure is proposed to solve a two-player zero-sum Markov games. Under suitable assumption, the boundedness of the proposed iterates is obtained theoretically. Using results from stochastic approximation, the almost sure convergence of the proposed two-step minimax Q-learning is obtained theoretically. More specifically, the proposed algorithm converges to the game theoretic optimal value with probability one, when the model information is not known. Numerical simulation authenticate that the proposed algorithm is effective and easy to implement.