GRASP: Gradient Realignment via Active Shared Perception for Multi-Agent Collaborative Optimization
This addresses non-stationarity issues in multi-agent systems for applications like gaming and robotics, representing a novel method rather than an incremental improvement.
The paper tackles the problem of non-stationarity in multi-agent reinforcement learning, which causes equilibrium oscillations and slow convergence, by proposing GRASP, a framework that uses active shared perception and a consensus gradient to achieve a generalized Bellman equilibrium, resulting in improved performance on benchmarks like StarCraft II and Google Research Football.
Non-stationarity arises from concurrent policy updates and leads to persistent environmental fluctuations. Existing approaches like Centralized Training with Decentralized Execution (CTDE) and sequential update schemes mitigate this issue. However, since the perception of the policies of other agents remains dependent on sampling environmental interaction data, the agent essentially operates in a passive perception state. This inevitably triggers equilibrium oscillations and significantly slows the convergence speed of the system. To address this issue, we propose Gradient Realignment via Active Shared Perception (GRASP), a novel framework that defines generalized Bellman equilibrium as a stable objective for policy evolution. The core mechanism of GRASP involves utilizing the independent gradients of agents to derive a defined consensus gradient, enabling agents to actively perceive policy updates and optimize team collaboration. Theoretically, we leverage the Kakutani Fixed-Point Theorem to prove that the consensus direction $u^*$ guarantees the existence and attainability of this equilibrium. Extensive experiments on StarCraft II Multi-Agent Challenge (SMAC) and Google Research Football (GRF) demonstrate the scalability and promising performance of the framework.