Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices
This work addresses the challenge of leveraging distributed offline data for RL in applications where online collection is infeasible, offering a solution that is communication-efficient and reduces data requirements, though it is incremental as it builds on existing offline RL methods.
The paper tackles the problem of federated offline reinforcement learning by proposing FedLCB-Q, a Q-learning variant that enables multiple agents to collaboratively learn from offline datasets without requiring high-quality data at each agent, achieving linear speedup in sample complexity and communication efficiency with a number of rounds linear in the horizon length.
Offline reinforcement learning (RL), which seeks to learn an optimal policy using offline data, has garnered significant interest due to its potential in critical applications where online data collection is infeasible or expensive. This work explores the benefit of federated learning for offline RL, aiming at collaboratively leveraging offline datasets at multiple agents. Focusing on finite-horizon episodic tabular Markov decision processes (MDPs), we design FedLCB-Q, a variant of the popular model-free Q-learning algorithm tailored for federated offline RL. FedLCB-Q updates local Q-functions at agents with novel learning rate schedules and aggregates them at a central server using importance averaging and a carefully designed pessimistic penalty term. Our sample complexity analysis reveals that, with appropriately chosen parameters and synchronization schedules, FedLCB-Q achieves linear speedup in terms of the number of agents without requiring high-quality datasets at individual agents, as long as the local datasets collectively cover the state-action space visited by the optimal policy, highlighting the power of collaboration in the federated setting. In fact, the sample complexity almost matches that of the single-agent counterpart, as if all the data are stored at a central location, up to polynomial factors of the horizon length. Furthermore, FedLCB-Q is communication-efficient, where the number of communication rounds is only linear with respect to the horizon length up to logarithmic factors.