Event-Based Communication in Distributed Q-Learning
This work addresses communication efficiency for distributed reinforcement learning systems, though it appears incremental as it adapts existing event-triggered control techniques to a known bottleneck.
The paper tackles the problem of high communication overhead in distributed Q-learning systems by proposing an event-based communication approach, which reduces data transmission rates while maintaining convergence guarantees comparable to vanilla Q-learning.
We present an approach to reduce the communication of information needed on a Distributed Q-Learning system inspired by Event Triggered Control (ETC) techniques. We consider a baseline scenario of a distributed Q-learning problem on a Markov Decision Process (MDP). Following an event-based approach, N agents explore the MDP and communicate experiences to a central learner only when necessary, which performs updates of the actor Q functions. We design an Event Based distributed Q learning system (EBd-Q), and derive convergence guarantees with respect to a vanilla Q-learning algorithm. We present experimental results showing that event-based communication results in a substantial reduction of data transmission rates in such distributed systems. Additionally, we discuss what effects (desired and undesired) these event-based approaches have on the learning processes studied, and how they can be applied to more complex multi-agent systems.