Holo-Captioning: Toward the Text Equivalent of 3D Scenes
For researchers in 3D scene understanding and multimodal AI, this work defines a new task and provides a strong baseline, though it is an initial step and may be incremental in terms of methodology.
This paper introduces holo-captioning, a new task of generating structured text descriptions for 3D scenes that include semantic tags, locations, attributes, and relations. The proposed HoloScribe model outperforms existing 3D dense captioners and LLM generalists on a new benchmark of over 15K scenes.
This work introduces holo-captioning, a novel task that strives to seek the text equivalent of 3D scenes. As the initial step, we formulate holo-captioning as generating a structured textual description that comprehensively depicts all entities within a 3D scene -- including their semantic tags, spatial locations, attributes, and inter-entity relations. To tackle this challenging task, we first develop an effective captioning engine to produce detailed descriptions of individual entity instances and instance pairs, and contribute a large-scale benchmark comprising over 15K scenes for training and evaluation. Building upon this foundation, we propose HoloScribe, a novel model that features an instance-aware decoupled pipeline for generating structured holo-captions, and further incorporates anchor-aware instance linking to identify relational instance pairs. Additionally, we propose a comprehensive evaluation metric named HoloScore, and provide a human-curated test set to ensure reliable model assessment. Experimental results demonstrate that HoloScribe significantly outperforms state-of-the-art 3D dense captioners and 3D LLM generalists, underscoring the effectiveness of our approach. Project page: https://visual-ai.github.io/holocap/