Nehal Baganal Krishna, Anam Tahir, Firas Khamis et al.
Asynchronous Distributed Reinforcement Learning (DRL) can suffer from degraded convergence when model updates become stale, often the result of network congestion and packet loss during large-scale training. This paper presents \textit{Olaf}, an in-network DRL accelerator engine, which \textit{opportunistically aggregates} asynchronously generated model updates \textit{on the fly} while they traverse the data plane queue. This aggregation operation motivates an alternative queue design, which we prototype and envision for future Top-of-Rack switches. We further present corresponding host-side transmission control in the face of possible congestion, taking advantage of in-network accelerator feedback. A quantification of model staleness, denoted Age-of-Model (AoM), together with a formal verifier, allows us to reason on system-wide AoM objectives in multi DRL-cluster scenarios. Olaf shows significant reductions in model staleness and queue congestion, improving overall convergence rate for asynchronous DRL workloads.