Byzantine-Robust Online and Offline Distributed Reinforcement Learning
This addresses the challenge of securing distributed RL systems against Byzantine attacks, which is critical for applications like multi-agent robotics or federated learning, though it builds incrementally on prior robust estimation methods.
The paper tackles the problem of distributed reinforcement learning with adversarial agents that can report arbitrary fake data, proposing algorithms for both offline and online settings that achieve near-optimal sample complexities and improved robustness guarantees.
We consider a distributed reinforcement learning setting where multiple agents separately explore the environment and communicate their experiences through a central server. However, $α$-fraction of agents are adversarial and can report arbitrary fake information. Critically, these adversarial agents can collude and their fake data can be of any sizes. We desire to robustly identify a near-optimal policy for the underlying Markov decision process in the presence of these adversarial agents. Our main technical contribution is Weighted-Clique, a novel algorithm for the robust mean estimation from batches problem, that can handle arbitrary batch sizes. Building upon this new estimator, in the offline setting, we design a Byzantine-robust distributed pessimistic value iteration algorithm; in the online setting, we design a Byzantine-robust distributed optimistic value iteration algorithm. Both algorithms obtain near-optimal sample complexities and achieve superior robustness guarantee than prior works.