Olaf: Programmable Data Plane Acceleration for Asynchronous Distributed Reinforcement Learning
For large-scale DRL training, Olaf addresses the bottleneck of network-induced model staleness, offering a programmable data plane solution to improve convergence.
Olaf is an in-network accelerator that aggregates asynchronous model updates on the fly to reduce model staleness and queue congestion, improving convergence rate for asynchronous distributed reinforcement learning.
Asynchronous Distributed Reinforcement Learning (DRL) can suffer from degraded convergence when model updates become stale, often the result of network congestion and packet loss during large-scale training. This paper presents \textit{Olaf}, an in-network DRL accelerator engine, which \textit{opportunistically aggregates} asynchronously generated model updates \textit{on the fly} while they traverse the data plane queue. This aggregation operation motivates an alternative queue design, which we prototype and envision for future Top-of-Rack switches. We further present corresponding host-side transmission control in the face of possible congestion, taking advantage of in-network accelerator feedback. A quantification of model staleness, denoted Age-of-Model (AoM), together with a formal verifier, allows us to reason on system-wide AoM objectives in multi DRL-cluster scenarios. Olaf shows significant reductions in model staleness and queue congestion, improving overall convergence rate for asynchronous DRL workloads.