Impact of Decentralized Learning on Player Utilities in Stackelberg Games
This work addresses the impact of decentralized learning on agent utilities in sequential interactions, such as recommender systems and chatbots, with incremental algorithmic improvements.
The paper tackled the problem of decentralized learning in two-agent Stackelberg games, showing that standard regret benchmarks lead to worst-case linear regret, and developed algorithms achieving near-optimal O(T^{2/3}) regret for both players, with faster O(sqrt(T)) learning possible under relaxed conditions.
When deployed in the world, a learning agent such as a recommender system or a chatbot often repeatedly interacts with another learning agent (such as a user) over time. In many such two-agent systems, each agent learns separately and the rewards of the two agents are not perfectly aligned. To better understand such cases, we examine the learning dynamics of the two-agent system and the implications for each agent's objective. We model these systems as Stackelberg games with decentralized learning and show that standard regret benchmarks (such as Stackelberg equilibrium payoffs) result in worst-case linear regret for at least one player. To better capture these systems, we construct a relaxed regret benchmark that is tolerant to small learning errors by agents. We show that standard learning algorithms fail to provide sublinear regret, and we develop algorithms to achieve near-optimal $O(T^{2/3})$ regret for both players with respect to these benchmarks. We further design relaxed environments under which faster learning ($O(\sqrt{T})$) is possible. Altogether, our results take a step towards assessing how two-agent interactions in sequential and decentralized learning environments affect the utility of both agents.