Personalized Observation Normalization for Federated Reinforcement Learning in Simulation Environments with Heterogeneity
For federated reinforcement learning in heterogeneous environments, this work addresses the challenge of non-identical input distributions and imbalanced updates, though the method is incremental.
Federated reinforcement learning struggles with heterogeneous environments due to differing state-transition dynamics. The proposed personalized observation normalization (PON) method, which normalizes state inputs locally per agent, accelerates training and achieves superior performance on heterogeneous MuJoCo tasks compared to baselines.
Federated reinforcement learning (FedRL) enables multiple agents to collaboratively train a global policy without sharing raw data, making it ideal for privacy-sensitive applications. However, FedRL faces challenges in heterogeneous environments where differing state-transition dynamics lead to non-identical input distributions and imbalanced parameter updates during aggregation. Therefore, this paper develops a personalized observation normalization (PON) method, allowing each agent to locally normalize raw state inputs using a continuously updated running mean and variance. This design ensures consistent scaling of local feature without overshadowing across agents during aggregation. Furthermore, we demonstrate that sharing normalization parameters across agents is ineffective due to the diverse local input distributions, which highlights the necessity of personalized statistics. Experiments on heterogeneous MuJoCo tasks show that our developed PON accelerates training and achieves superior performance compared to baseline methods.