OmniSpace: Efficient Geometry Awareness for Autonomous Vehicles MLLMs
This work addresses the bottleneck of weak cross-view correspondence and depth estimation in MLLMs for autonomous driving, providing a simple yet effective method that avoids complex 3D pipelines.
OmniSpace introduces a plug-and-play paradigm that enhances geometry-aware spatial reasoning in MLLMs for autonomous vehicles using only 2D observations, achieving state-of-the-art performance on planning (nuScenes, Bench2Drive), risk detection (nuInstruct), language (Omnidrive), and generalization (DriveBench) benchmarks.
Multimodal Large Language Models (MLLMs) have achieved remarkable performance on 2D visual tasks, yet enhancing their spatial intelligence for real-world applications such as Autonomous Vehicles (AV) remains an open challenge. Existing geometry-aware MLLMs typically rely on auxiliary 3D models at inference time, introducing pipeline complexity and the risk of cascading failures. In this paper, we present OmniSpace, a simple yet effective plug-and-play paradigm for geometry-aware spatial reasoning from purely 2D observations. Motivated by our finding that current MLLMs are bottlenecked by weak cross-view correspondence and depth estimation, OmniSpace introduces a Camera Pose Injector, a Multi-view Epipolar Attention module, and a 3D Geometric Distillation objective that jointly address these two limitations by transferring geometric knowledge into the model. Extensive experiments show that OmniSpace surpasses existing methods on planning benchmarks (nuScenes, Bench2Drive), risk detection (nuInstruct), language (Omnidrive), and generalization (DriveBench).