Decoupled Delay Compensation: Enhancing Pre-trained MARL Policies via Learned Dynamics Filtering
For practitioners deploying pre-trained MARL policies in real-world systems with communication imperfections, this work offers a plug-in solution to mitigate performance loss without retraining.
The paper addresses performance degradation in multi-agent reinforcement learning (MARL) under communication delays and packet loss by proposing a modular state-estimation layer that replaces delayed observations with belief-state estimates. The method consistently improves robustness across diverse benchmarks, with significant gains in coordination-intensive tasks.
Real-world multi-agent reinforcement learning (MARL) systems must often operate under stale observations, stochastic communication delays, and intermittent packet loss. Policies trained under idealized synchronous conditions frequently exhibit significant performance degradation in these regimes because they act on outdated feedback. We propose a modular execution-stage state-estimation layer that replaces delayed communicated observations with current belief-state estimates. The framework integrates a learned Gated transition model with a recursive Kalman filtering layer to estimate instantaneous states from asynchronous measurements. A primary advantage of this approach is its modularity, The estimator serves as a plug-in for pre-trained policies, requiring no modifications to the original MARL training algorithm, architecture, or reward structure. Evaluation across diverse multi-agent and continuous-control benchmarks demonstrates that the proposed layer consistently enhances robustness to communication latency and message loss. The most significant performance gains are observed in coordination-intensive and dynamically unstable tasks where temporal consistency is critical for control.