Distilling 3D Spatial Reasoning into a Lightweight Vision-Language Model with CoT
Enables efficient 3D scene understanding on resource-constrained platforms by compressing large VLMs while preserving spatial reasoning capabilities.
The paper distills spatial reasoning from a 7B 3D VLM teacher into a 2.29B student, achieving 8.7x lower latency and 3x size reduction while retaining 54-72% of teacher performance. The student uses a novel Hidden CoT mechanism with learnable latent tokens to improve reasoning without explicit CoT data.
Large-scale 3D vision-language models (VLMs) like LLaVA-3D offer strong spatial reasoning but are difficult to deploy due to high computational costs. We propose a knowledge distillation framework that transfers spatial reasoning from a 7B teacher to a 2.29B student model. Our approach achieves 8.7x lower inference latency and a 3x reduction in model size while retaining 54-72% of the teacher's performance. The framework utilizes VGGT as the vision encoder and a multi-task distillation pipeline with uncertainty-aware loss weighting. To improve reasoning without chain-of-thought (CoT) data, we introduce "Hidden CoT": learnable latent tokens that serve as an internal scratchpad before answer generation. This is the first use of latent scratchpad reasoning in distilled 3D VLMs. The student model jointly performs spatial description, depth estimation, and object detection. Experiments on ScanNet and 3D-FRONT show strong spatial understanding, reaching 68-72% accuracy on proximity and contact tasks. Our framework enables efficient 3D scene QA on resource-constrained platforms.