Opponent Modeling in Deep Reinforcement Learning
This work addresses the challenge of adapting to dynamic opponents in multi-agent systems, offering a domain-specific improvement for applications like gaming and simulation.
The paper tackles opponent modeling in multi-agent deep reinforcement learning by introducing neural-based models that jointly learn a policy and opponent behavior using a Mixture-of-Experts architecture, achieving superior performance over DQN and its variants in simulated soccer and trivia games.
Opponent modeling is necessary in multi-agent settings where secondary agents with competing goals also adapt their strategies, yet it remains challenging because strategies interact with each other and change. Most previous work focuses on developing probabilistic models or parameterized strategies for specific applications. Inspired by the recent success of deep reinforcement learning, we present neural-based models that jointly learn a policy and the behavior of opponents. Instead of explicitly predicting the opponent's action, we encode observation of the opponents into a deep Q-Network (DQN); however, we retain explicit modeling (if desired) using multitasking. By using a Mixture-of-Experts architecture, our model automatically discovers different strategy patterns of opponents without extra supervision. We evaluate our models on a simulated soccer game and a popular trivia game, showing superior performance over DQN and its variants.