Policy Iteration for Pareto-Optimal Policies in Stochastic Stackelberg Games
This addresses convergence challenges in multi-agent reinforcement learning for stochastic Stackelberg games, offering a more robust solution, though it appears incremental as it builds on existing policy iteration methods.
The paper tackled the problem of non-existence and convergence issues in stationary Stackelberg equilibria for stochastic games by introducing Pareto-optimal policies as an alternative, proposing a policy iteration algorithm with proven monotone improvement and convergence, including to SSEs in special cases.
In general-sum stochastic games, a stationary Stackelberg equilibrium (SSE) does not always exist, in which the leader maximizes leader's return for all the initial states when the follower takes the best response against the leader's policy. Existing methods of determining the SSEs require strong assumptions to guarantee the convergence and the coincidence of the limit with the SSE. Moreover, our analysis suggests that the performance at the fixed points of these methods is not reasonable when they are not SSEs. Herein, we introduced the concept of Pareto-optimality as a reasonable alternative to SSEs. We derive the policy improvement theorem for stochastic games with the best-response follower and propose an iterative algorithm to determine the Pareto-optimal policies based on it. Monotone improvement and convergence of the proposed approach are proved, and its convergence to SSEs is proved in a special case.