Metric Policy Representations for Opponent Modeling
This addresses the challenge of opponent modeling for agents in multi-agent environments, though it appears incremental as it builds on existing representation learning approaches.
The paper tackles the problem of non-stationarity in multi-agent reinforcement learning by proposing a method to learn policy representations that capture similarities and differences between agents' policies, enabling better generalization to unseen agents in three tasks.
In multi-agent reinforcement learning, the inherent non-stationarity of the environment caused by other agents' actions posed significant difficulties for an agent to learn a good policy independently. One way to deal with non-stationarity is opponent modeling, by which the agent takes into consideration the influence of other agents' policies. Most existing work relies on predicting other agents' actions or goals, or discriminating between different policies. However, such modeling fails to capture the similarities and differences between policies simultaneously and thus cannot provide enough useful information when generalizing to unseen agents. To address this, we propose a general method to learn representations of other agents' policies, such that the distance between policies is deliberately reflected by the distance between representations, while the policy distance is inferred from the sampled joint action distributions during training. We empirically show that the agent conditioned on the learned policy representation can well generalize to unseen agents in three multi-agent tasks.