Markov Potential Game and Multi-Agent Reinforcement Learning for Autonomous Driving
This work addresses the problem of reliable multi-agent decision-making for autonomous vehicles, offering an incremental improvement by providing construction rules for MPGs in this domain.
The paper tackled the challenge of constructing Markov potential games (MPGs) for autonomous driving to ensure Nash equilibrium attainability, showing that their conditions accommodate general driving objectives and evaluating the policy in simulations and real traffic datasets with comparative studies against single-agent RL and human drivers.
Autonomous driving (AD) requires safe and reliable decision-making among interacting agents, e.g., vehicles, bicycles, and pedestrians. Multi-agent reinforcement learning (MARL) modeled by Markov games (MGs) provides a suitable framework to characterize such agents' interactions during decision-making. Nash equilibria (NEs) are often the desired solution in an MG. However, it is typically challenging to compute an NE in general-sum games, unless the game is a Markov potential game (MPG), which ensures the NE attainability under a few learning algorithms such as gradient play. However, it has been an open question how to construct an MPG and whether these construction rules are suitable for AD applications. In this paper, we provide sufficient conditions under which an MG is an MPG and show that these conditions can accommodate general driving objectives for autonomous vehicles (AVs) using highway forced merge scenarios as illustrative examples. A parameter-sharing neural network (NN) structure is designed to enable decentralized policy execution. The trained driving policy from MPGs is evaluated in both simulated and naturalistic traffic datasets. Comparative studies with single-agent RL and with human drivers whose behaviors are recorded in the traffic datasets are reported, respectively.