LLaVA$^3$: Representing 3D Scenes like a Cubist Painter to Boost 3D Scene Understanding of VLMs
This addresses the problem of limited 3D training data for VLMs, offering a domain-specific improvement for 3D scene understanding.
The paper tackles the challenge of 3D scene understanding for vision-language models (VLMs) by introducing LLaVA$^3$, which uses multi-view 2D images to create omnidirectional visual representations of objects without fine-tuning, and shows it outperforms previous 2D-based VLM solutions on 3D VQA and 3D language grounding tasks.
Developing a multi-modal language model capable of understanding 3D scenes remains challenging due to the limited availability of 3D training data, in contrast to the abundance of 2D datasets used for vision-language models (VLM). As an alternative, we introduce LLaVA$^3$ (pronounced LLaVA-Cube), a novel method that improves the 3D scene understanding capabilities of VLM using only multi-view 2D images and without any fine-tuning. Inspired by Cubist painters, who represented multiple viewpoints of a 3D object within a single picture, we propose to describe the 3D scene for the VLM through omnidirectional visual representations of each object. These representations are derived from an intermediate multi-view 3D reconstruction of the scene. Extensive experiments on 3D VQA and 3D language grounding show that our approach outperforms previous 2D-based VLM solutions.