From Imitation to Optimization: A Comparative Study of Offline Learning for Autonomous Driving
This addresses the challenge of safe and practical autonomous driving by demonstrating a significant improvement over imitation learning, though it is incremental as it applies an existing offline RL method to a specific domain.
The paper tackled the problem of learning robust driving policies from static expert data, showing that applying Conservative Q-Learning (CQL) to a Transformer-based architecture achieved a 3.2x higher success rate and 7.4x lower collision rate compared to Behavioral Cloning baselines in evaluations on the Waymo Open Motion Dataset.
Learning robust driving policies from large-scale, real-world datasets is a central challenge in autonomous driving, as online data collection is often unsafe and impractical. While Behavioral Cloning (BC) offers a straightforward approach to imitation learning, policies trained with BC are notoriously brittle and suffer from compounding errors in closed-loop execution. This work presents a comprehensive pipeline and a comparative study to address this limitation. We first develop a series of increasingly sophisticated BC baselines, culminating in a Transformer-based model that operates on a structured, entity-centric state representation. While this model achieves low imitation loss, we show that it still fails in long-horizon simulations. We then demonstrate that by applying a state-of-the-art Offline Reinforcement Learning algorithm, Conservative Q-Learning (CQL), to the same data and architecture, we can learn a significantly more robust policy. Using a carefully engineered reward function, the CQL agent learns a conservative value function that enables it to recover from minor errors and avoid out-of-distribution states. In a large-scale evaluation on 1,000 unseen scenarios from the Waymo Open Motion Dataset, our final CQL agent achieves a 3.2x higher success rate and a 7.4x lower collision rate than the strongest BC baseline, proving that an offline RL approach is critical for learning robust, long-horizon driving policies from static expert data.