A Comparison of Self-Play Algorithms Under a Generalized Framework
This work provides a formal foundation for comparing self-play algorithms, which is incremental as it builds on existing methods without introducing new paradigms.
The authors tackled the lack of a formal model for self-play in multiagent reinforcement learning by developing a generalized framework to compare algorithms, finding that various self-play definitions lead to cyclic policy evolutions during training.
Throughout scientific history, overarching theoretical frameworks have allowed researchers to grow beyond personal intuitions and culturally biased theories. They allow to verify and replicate existing findings, and to link is connected results. The notion of self-play, albeit often cited in multiagent Reinforcement Learning, has never been grounded in a formal model. We present a formalized framework, with clearly defined assumptions, which encapsulates the meaning of self-play as abstracted from various existing self-play algorithms. This framework is framed as an approximation to a theoretical solution concept for multiagent training. On a simple environment, we qualitatively measure how well a subset of the captured self-play methods approximate this solution when paired with the famous PPO algorithm. We also provide insights on interpreting quantitative metrics of performance for self-play training. Our results indicate that, throughout training, various self-play definitions exhibit cyclic policy evolutions.