An Encoded Corrective Double Deep Q-Networks for Multi-Agent Control Systems
For researchers in multi-agent reinforcement learning, this work provides a method to handle realistic communication imperfections (noise, delay, asynchrony) that are often ignored, though the improvements are incremental over existing actor-critic approaches.
This paper addresses the problem of synthesizing control policies for heterogeneous multi-agent systems with noisy and delayed communication. The proposed encoded corrective double deep Q-network framework outperforms baselines across multiple test cases, with numerical regret analysis confirming its effectiveness.
This paper studies the synthesis of control policies for heterogeneous and interconnected multi-agent systems that collaborate through data exchange over a communication network to minimize a collective cost. We propose a distributed encoded corrective double actor-critic framework that integrates a novel message-passing mechanism. Existing methods assume noise-free and delay-free access to the global or partial states and overlook the fact that the global states, though noisy and delayed, can be progressively reconstructed and refined over time. In contrast, this work explicitly models communication sampling asynchrony, delay, and link noise based on the network configuration. The proposed message-passing mechanism characterizes timing and information flow to refine and time shift global state information, which is then used to incrementally correct the Q-networks. The double Q-network design mitigates overestimation bias, while the shared encoder coupling the actor-critic networks captures inter-agent dependencies. We evaluate our approach in multiple test cases, demonstrate its effectiveness over various baselines, and provide a numerical regret analysis.