Autonomous Vehicle Controllers From End-to-End Differentiable Simulation
This work addresses the challenge of learning robust and generalizable controllers for autonomous vehicles, offering a more efficient and prior-informed alternative to existing methods, though it is incremental as it builds on differentiable simulation techniques.
The authors tackled the problem of poor generalization in autonomous vehicle controllers trained via behavioral cloning by using a differentiable simulator to enable end-to-end training with analytic policy gradients, resulting in significant improvements in performance, robustness to noise, and more human-like handling compared to behavioral cloning.
Current methods to learn controllers for autonomous vehicles (AVs) focus on behavioural cloning. Being trained only on exact historic data, the resulting agents often generalize poorly to novel scenarios. Simulators provide the opportunity to go beyond offline datasets, but they are still treated as complicated black boxes, only used to update the global simulation state. As a result, these RL algorithms are slow, sample-inefficient, and prior-agnostic. In this work, we leverage a differentiable simulator and design an analytic policy gradients (APG) approach to training AV controllers on the large-scale Waymo Open Motion Dataset. Our proposed framework brings the differentiable simulator into an end-to-end training loop, where gradients of the environment dynamics serve as a useful prior to help the agent learn a more grounded policy. We combine this setup with a recurrent architecture that can efficiently propagate temporal information across long simulated trajectories. This APG method allows us to learn robust, accurate, and fast policies, while only requiring widely-available expert trajectories, instead of scarce expert actions. We compare to behavioural cloning and find significant improvements in performance and robustness to noise in the dynamics, as well as overall more intuitive human-like handling.