Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Models
This work solves the exploration bottleneck in autonomous driving VLA models, which is an incremental improvement over existing methods.
The paper tackles the Narrow Policy limitation in autonomous Vision-Language-Action (VLA) models, where Imitation Learning restricts exploration and hinders Reinforcement Learning performance, and proposes Curious-VLA to address this, achieving state-of-the-art results on the Navsim benchmark with PDMS 90.3 and EPDMS 85.4.
We identify a fundamental Narrow Policy limitation undermining the performance of autonomous VLA models, where driving Imitation Learning (IL) tends to collapse exploration and limit the potential of subsequent Reinforcement Learning (RL) stages, which often saturate prematurely due to insufficient feedback diversity. Thereby, we propose Curious-VLA, a framework that alleviates the exploit-explore dilemma through a two-stage design. During IL, we introduce a Feasible Trajectory Expansion (FTE) strategy to generate multiple physically valid trajectories and a step-wise normalized trajectory representation to adapt this diverse data. In the RL stage, we present Adaptive Diversity-Aware Sampling (ADAS) that prioritizes high-diversity samples and introduce Spanning Driving Reward (SDR) with a focal style weighting to amplify reward's value span for improving sensitivity to driving quality. On the Navsim benchmark, Curious-VLA achieves SoTA results (PDMS 90.3, EPDMS 85.4) and a Best-of-N PDMS of 94.8, demonstrating its effectiveness in unlocking the exploratory potential of VLA models. Code: https://github.com/Mashiroln/curious_vla.git.