CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning
This addresses multi-goal multi-agent reinforcement learning, which is incremental as it builds on prior work by restructuring the problem and adding credit assignment for improved efficiency.
The paper tackled the problem of cooperative multi-agent control with individual goals by introducing a two-stage curriculum and a new policy gradient with credit assignment, resulting in significantly faster learning compared to existing methods on navigation, traffic, and game tasks.
A variety of cooperative multi-agent control problems require agents to achieve individual goals while contributing to collective success. This multi-goal multi-agent setting poses difficulties for recent algorithms, which primarily target settings with a single global reward, due to two new challenges: efficient exploration for learning both individual goal attainment and cooperation for others' success, and credit-assignment for interactions between actions and goals of different agents. To address both challenges, we restructure the problem into a novel two-stage curriculum, in which single-agent goal attainment is learned prior to learning multi-agent cooperation, and we derive a new multi-goal multi-agent policy gradient with a credit function for localized credit assignment. We use a function augmentation scheme to bridge value and policy functions across the curriculum. The complete architecture, called CM3, learns significantly faster than direct adaptations of existing algorithms on three challenging multi-goal multi-agent problems: cooperative navigation in difficult formations, negotiating multi-vehicle lane changes in the SUMO traffic simulator, and strategic cooperation in a Checkers environment.