FAST: A Framework for Aligned Sampling and Training in Parallel Reinforcement Learning for Autonomous Driving
For autonomous driving researchers using parallel reinforcement learning, FAST provides a method to eliminate sampling inefficiency caused by straggler effects without sacrificing data diversity or theoretical consistency.
FAST addresses the straggler effect in parallel reinforcement learning for autonomous driving by using Dynamic Parallel Sampling Alignment (DPSA) and Scaled Mask-Padding Optimization (SMPO), achieving at least a 1.78x wall-clock speedup over single-clip baseline while preserving statistical unbiasedness.
Deep reinforcement learning is pivotal for closed-loop autonomous driving yet remains constrained by severe bottlenecks in sampling efficiency. Standard parallel sampling mitigates this but suffers from the straggler effect, where the premature termination of a single environment necessitates a synchronized batch re-initialization, leading to suboptimal sample utilization and prohibitive re-initialization latency. To address this, we propose FAST, a synchronous parallel framework tailored for closed-loop simulation. Specifically, FAST employs Dynamic Parallel Sampling Alignment (DPSA) to maintain vectorization synchronization by extending terminated episodes via virtual continuation, thereby decoupling the sampling loop from individual terminations. By dynamically triggering global truncation based on the termination rate of parallel clips, FAST effectively eliminates the bottleneck of premature resets without sacrificing data diversity. Furthermore, to strictly preserve theoretical consistency, we incorporate a Scaled Mask-Padding Optimization (SMPO) that leverages validity masking and adaptive loss normalization to nullify the bias from auxiliary padding data. Empirical evaluations demonstrate that FAST achieves at least a 1.78 times wall-clock speedup over the single-clip baseline while preserving statistical unbiasedness.