Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updates
This addresses a critical bottleneck in federated reinforcement learning for agents collaborating without sharing data, though it is incremental as it builds on existing ADMM and NPG techniques.
The paper tackles the high communication overhead in federated natural policy gradient methods by proposing FedNPG-ADMM, which reduces communication complexity from O(d^2) to O(d) per iteration while maintaining the same convergence rate and reward performance as standard methods.
Federated reinforcement learning (FedRL) enables agents to collaboratively train a global policy without sharing their individual data. However, high communication overhead remains a critical bottleneck, particularly for natural policy gradient (NPG) methods, which are second-order. To address this issue, we propose the FedNPG-ADMM framework, which leverages the alternating direction method of multipliers (ADMM) to approximate global NPG directions efficiently. We theoretically demonstrate that using ADMM-based gradient updates reduces communication complexity from ${O}({d^{2}})$ to ${O}({d})$ at each iteration, where $d$ is the number of model parameters. Furthermore, we show that achieving an $ε$-error stationary convergence requires ${O}(\frac{1}{(1-γ)^{2}ε})$ iterations for discount factor $γ$, demonstrating that FedNPG-ADMM maintains the same convergence rate as the standard FedNPG. Through evaluation of the proposed algorithms in MuJoCo environments, we demonstrate that FedNPG-ADMM maintains the reward performance of standard FedNPG, and that its convergence rate improves when the number of federated agents increases.