VLADriveBench: Evaluating CoT-Action Relationship in VLA for Autonomous Driving
For autonomous driving researchers, this work provides a framework to assess CoT-action relationships, revealing that current metrics can be misleading.
Existing benchmarks for vision-language-action models evaluate only trajectory quality, not whether chain-of-thought reasoning is relevant or causally connected to driving actions. VLADriveBench introduces observational metrics and a CoT intervention protocol, revealing that high observational alignment can coexist with epiphenomenal CoT, while lower-scoring models may have strongly causal CoT.
Vision-language-action (VLA) models generate chain-of-thought (CoT) reasoning alongside driving trajectories, but existing benchmarks evaluate only trajectory quality and do not assess whether the CoT is relevant, consistent, or causally connected to the driving action. We introduce VLADriveBench, a framework that combines observational metrics (mentioning, hallucination, contradiction, action alignment) with a CoT intervention protocol to provide complementary views of the CoT-action relationship. Applying VLADriveBench to three models across two architectures, we find that the two analyses can diverge sharply: ORION scores highest on observational alignment yet its CoT is epiphenomenal, while Alpamayo v1.5 scores lower yet its CoT is strongly causal, with visual salience gating the extent of CoT influence.