PointLLM-R: Enhancing 3D Point Cloud Reasoning via Chain-of-Thought
For researchers in 3D multimodal learning, this work introduces a method to incorporate explicit reasoning into 3D point cloud understanding, addressing a gap in existing models.
The authors propose a data-centric framework to construct large-scale Chain-of-Thought (CoT) supervision for 3D point cloud understanding, resulting in a dataset PoCoTI (55K samples) and a model PointLLM-R. PointLLM-R achieves state-of-the-art performance on generative 3D classification and captioning tasks, and generalizes to real-world scanned point clouds and multi-turn dialogues.
Understanding 3D point clouds through language remains a fundamental challenge in computer graphics and visual computing, due to the irregular structure of point cloud data and the lack of explicit reasoning in existing 3D multimodal models. While Chain-of-Thought (CoT) reasoning has shown strong effectiveness in LLMs and image-based MLLMs, its extension to 3D understanding remains largely underexplored. In this paper, we propose a data-centric framework for constructing large-scale CoT supervision tailored to 3D point cloud understanding. Our framework consists of a two-stage pipeline that first refines point-text instruction data via vision-language-model-based quality evaluation and reference-guided refinement, and then synthesizes high-quality reasoning paths through Human-in-the-Loop Prompt Optimization (HiLPO). Using this approach, we build PoCoTI, a CoT-enhanced point-text instruction-following dataset containing 55K samples with explicit reasoning paths. Fine-tuning PointLLM on PoCoTI yields PointLLM-R, a reasoning-capable 3D multimodal language model. Extensive experiments on generative 3D classification and captioning demonstrate that PointLLM-R achieves state-of-the-art performance and generalizes robustly to real-world scanned point clouds and multi-turn dialogue scenarios.