Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentation
This work addresses a domain-specific problem for researchers in 3D vision and multimodal AI, offering incremental improvements in handling complex spatial instructions.
The paper tackles the challenge of enhancing spatial reasoning in multimodal large language models for 3D point cloud perception by proposing a reasoning-based segmentation framework (R$^2$S) and a new dataset (3D ReasonSeg), resulting in stronger spatial reasoning capabilities as demonstrated in experiments.
Recent advances in point cloud perception have demonstrated remarkable progress in scene understanding through vision-language alignment leveraging large language models (LLMs). However, existing methods may still encounter challenges in handling complex instructions that require accurate spatial reasoning, even if the 3D point cloud data provides detailed spatial cues such as size and position for identifying the targets. To tackle this issue, we propose Relevant Reasoning Segmentation (R$^2$S), a reasoning-based segmentation framework. The framework emulates human cognitive processes by decomposing spatial reasoning into two sequential stages: first identifying relevant elements, then processing instructions guided by their associated visual priors. Furthermore, acknowledging the inadequacy of existing datasets in complex reasoning tasks, we introduce 3D ReasonSeg, a reasoning-based segmentation dataset comprising 25,185 training samples and 3,966 validation samples with precise annotations. Both quantitative and qualitative experiments demonstrate that the R$^2$S and 3D ReasonSeg effectively endow 3D point cloud perception with stronger spatial reasoning capabilities, and we hope that they can serve as a new baseline and benchmark for future work.