Over-the-air Federated Policy Gradient
This work addresses communication efficiency in distributed reinforcement learning for wireless networks, but appears incremental as it extends over-the-air methods to policy gradients.
The paper tackled the problem of federated policy gradient learning in wireless environments by proposing an over-the-air aggregation algorithm, and established convergence complexities for finding an ε-approximate stationary point under noise and channel distortion, with simulation results demonstrating effectiveness.
In recent years, over-the-air aggregation has been widely considered in large-scale distributed learning, optimization, and sensing. In this paper, we propose the over-the-air federated policy gradient algorithm, where all agents simultaneously broadcast an analog signal carrying local information to a common wireless channel, and a central controller uses the received aggregated waveform to update the policy parameters. We investigate the effect of noise and channel distortion on the convergence of the proposed algorithm, and establish the complexities of communication and sampling for finding an $ε$-approximate stationary point. Finally, we present some simulation results to show the effectiveness of the algorithm.