Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning
This addresses credit assignment for multi-agent systems with characteristics like health and partial observability, but is incremental as it builds on existing policy gradient methods.
The paper tackles the problem of credit assignment in multi-agent reinforcement learning by defining system health and incorporating it into policy gradients, resulting in significant learning performance improvements in particle and multiwalker robot environments.
This paper proposes a definition of system health in the context of multiple agents optimizing a joint reward function. We use this definition as a credit assignment term in a policy gradient algorithm to distinguish the contributions of individual agents to the global reward. The health-informed credit assignment is then extended to a multi-agent variant of the proximal policy optimization algorithm and demonstrated on particle and multiwalker robot environments that have characteristics such as system health, risk-taking, semi-expendable agents, continuous action spaces, and partial observability. We show significant improvement in learning performance compared to policy gradient methods that do not perform multi-agent credit assignment.