CorrectAD: A Self-Correcting Agentic System to Improve End-to-end Planning in Autonomous Driving
This addresses safety-critical failures in autonomous driving systems, offering a model-agnostic solution that is incremental by integrating existing methods like diffusion models and 3D layouts.
The paper tackles the long-tail problem in end-to-end autonomous driving planning by proposing CorrectAD, a self-correcting agentic system that uses generative models to simulate and correct failure cases, resulting in correction of 62.5% and 49.8% of failures and reducing collision rates by 39% and 27% on two datasets.
End-to-end planning methods are the de facto standard of the current autonomous driving system, while the robustness of the data-driven approaches suffers due to the notorious long-tail problem (i.e., rare but safety-critical failure cases). In this work, we explore whether recent diffusion-based video generation methods (a.k.a. world models), paired with structured 3D layouts, can enable a fully automated pipeline to self-correct such failure cases. We first introduce an agent to simulate the role of product manager, dubbed PM-Agent, which formulates data requirements to collect data similar to the failure cases. Then, we use a generative model that can simulate both data collection and annotation. However, existing generative models struggle to generate high-fidelity data conditioned on 3D layouts. To address this, we propose DriveSora, which can generate spatiotemporally consistent videos aligned with the 3D annotations requested by PM-Agent. We integrate these components into our self-correcting agentic system, CorrectAD. Importantly, our pipeline is an end-to-end model-agnostic and can be applied to improve any end-to-end planner. Evaluated on both nuScenes and a more challenging in-house dataset across multiple end-to-end planners, CorrectAD corrects 62.5% and 49.8% of failure cases, reducing collision rates by 39% and 27%, respectively.