FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectification
This addresses policy pollution in offline FRL for Internet-of-Things systems, offering a method to improve robustness with theoretical guarantees, though it is incremental as it builds on existing offline RL and federated learning techniques.
The paper tackles the problem of offline federated reinforcement learning (FRL) breaking down under low-quality, heterogeneous data by proposing FORLER, which combines Q-ensemble aggregation on the server with actor rectification on devices, and it shows consistent outperformance over baselines in experiments.
In Internet-of-Things systems, federated learning has advanced online reinforcement learning (RL) by enabling parallel policy training without sharing raw data. However, interacting with real environments online can be risky and costly, motivating offline federated RL (FRL), where local devices learn from fixed datasets. Despite its promise, offline FRL may break down under low-quality, heterogeneous data. Offline RL tends to get stuck in local optima, and in FRL, one device's suboptimal policy can degrade the aggregated model, i.e., policy pollution. We present FORLER, combining Q-ensemble aggregation on the server with actor rectification on devices. The server robustly merges device Q-functions to curb policy pollution and shift heavy computation off resource-constrained hardware without compromising privacy. Locally, actor rectification enriches policy gradients via a zeroth-order search for high-Q actions plus a bespoke regularizer that nudges the policy toward them. A $δ$-periodic strategy further reduces local computation. We theoretically provide safe policy improvement performance guarantees. Extensive experiments show FORLER consistently outperforms strong baselines under varying data quality and heterogeneity.