Improving 3D Labeling in Self-Driving by Inferring Vehicle Information using Vision Language Models
This work addresses the problem of efficient and accurate 3D vehicle labeling for autonomous driving, offering a practical improvement over manual labeling but is incremental in nature.
The authors propose using Vision Language Models (VLMs) to infer vehicle make, model, and generation from image crops, and output accurate 3D bounding box dimensions to seed manual labeling for self-driving applications. Their approach reduces manual labeling time and increases label quality, particularly in cases of significant vehicle occlusion.
We present an approach to improve 3D vehicle labeling in self-driving applications through zero-shot inference of vehicle information, leveraging Vehicle Make and Model Recognition (VMMR) methods. The proposed approach utilizes a Vision Language Model (VLM) to both infer a vehicle's make, model, and generation from image crops, and output accurate 3D bounding box dimensions to seed manual labeling. We evaluate the impact of iterative prompt engineering and the choice of different VLMs on both vehicle bounding box inference and make/model/generation recognition. When compared to strong baselines, the proposed approach not only shows high accuracy, but also excels in mitigating specific failure modes where VLMs provide better dimensions than initial lidar-aided human annotated labels (e.g., in cases of significant vehicle occlusion). Experiments on both public and proprietary data strongly suggest that our conclusions are generalizable across different labelers and datasets. The results demonstrate that integrating VLMs into the labeling process can reduce manual labeling time while increasing label quality.