3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding
This work addresses the problem of 3D city-scale perception and understanding for urban intelligence applications, representing a novel method for a known bottleneck rather than an incremental improvement.
The paper tackles the challenge of scaling multi-modality large language models to 3D city-scale environments by proposing 3DCity-LLM, a unified framework with a coarse-to-fine feature encoding strategy and a new 1.2 million sample dataset, achieving significant outperformance over state-of-the-art methods on two benchmarks.
While multi-modality large language models excel in object-centric or indoor scenarios, scaling them to 3D city-scale environments remains a formidable challenge. To bridge this gap, we propose 3DCity-LLM, a unified framework designed for 3D city-scale vision-language perception and understanding. 3DCity-LLM employs a coarse-to-fine feature encoding strategy comprising three parallel branches for target object, inter-object relationship, and global scene. To facilitate large-scale training, we introduce 3DCity-LLM-1.2M dataset that comprises approximately 1.2 million high-quality samples across seven representative task categories, ranging from fine-grained object analysis to multi-faceted scene planning. This strictly quality-controlled dataset integrates explicit 3D numerical information and diverse user-oriented simulations, enriching the question-answering diversity and realism of urban scenarios. Furthermore, we apply a multi-dimensional protocol based on text-similarity metrics and LLM-based semantic assessment to ensure faithful and comprehensive evaluations for all methods. Extensive experiments on two benchmarks demonstrate that 3DCity-LLM significantly outperforms existing state-of-the-art methods, offering a promising and meaningful direction for advancing spatial reasoning and urban intelligence. The source code and dataset are available at https://github.com/SYSU-3DSTAILab/3D-City-LLM.