Federated Ensemble-Directed Offline Reinforcement Learning
This addresses the problem of collaborative policy learning in distributed settings with limited data for applications like robotics, though it appears incremental as it combines existing techniques like ensemble learning with federated and offline RL.
The paper tackles federated offline reinforcement learning, where distributed agents learn a control policy from small pre-collected datasets, and shows that their FEDORA algorithm significantly outperforms other approaches in complex continuous control environments and real-world datasets, including on a mobile robot.
We consider the problem of federated offline reinforcement learning (RL), a scenario under which distributed learning agents must collaboratively learn a high-quality control policy only using small pre-collected datasets generated according to different unknown behavior policies. Naïvely combining a standard offline RL approach with a standard federated learning approach to solve this problem can lead to poorly performing policies. In response, we develop the Federated Ensemble-Directed Offline Reinforcement Learning Algorithm (FEDORA), which distills the collective wisdom of the clients using an ensemble learning approach. We develop the FEDORA codebase to utilize distributed compute resources on a federated learning platform. We show that FEDORA significantly outperforms other approaches, including offline RL over the combined data pool, in various complex continuous control environments and real-world datasets. Finally, we demonstrate the performance of FEDORA in the real-world on a mobile robot. We provide our code and a video of our experiments at \url{https://github.com/DesikRengarajan/FEDORA}.