The Actor-Critic Update Order Matters for PPO in Federated Reinforcement Learning
This addresses convergence issues in FRL for applications like autonomous driving, but it is incremental as it modifies an existing method (PPO) for a specific bottleneck.
The paper tackled the problem of data heterogeneity in Federated Reinforcement Learning (FRL) when using Proximal Policy Optimization (PPO), by proposing FedRAC which reverses the actor-critic update order to eliminate critic divergence. The result showed that FedRAC achieved higher cumulative rewards and faster convergence in five experiments, including a highly heterogeneous autonomous driving scenario.
In the context of Federated Reinforcement Learning (FRL), applying Proximal Policy Optimization (PPO) faces challenges related to the update order of its actor and critic due to the aggregation step occurring between successive iterations. In particular, when local actors are updated based on local critic estimations, the algorithm becomes vulnerable to data heterogeneity. As a result, the conventional update order in PPO (critic first, then actor) may cause heterogeneous gradient directions among clients, hindering convergence to a globally optimal policy. To address this issue, we propose FedRAC, which reverses the update order (actor first, then critic) to eliminate the divergence of critics from different clients. Theoretical analysis shows that the convergence bound of FedRAC is immune to data heterogeneity under mild conditions, i.e., bounded level of heterogeneity and accurate policy evaluation. Empirical results indicate that the proposed algorithm obtains higher cumulative rewards and converges more rapidly in five experiments, including three classical RL environments and a highly heterogeneous autonomous driving scenario using the SUMO traffic simulator.