OpenCity3D: What do Vision-Language Models know about Urban Environments?
This work addresses urban analytics for planning and policy by establishing a new paradigm for language-driven analysis, though it is incremental in expanding VLMs from indoor/autonomous driving to urban contexts.
The authors tackled the problem of applying vision-language models to urban-scale environments for high-level tasks like population density estimation and crime rate assessment, achieving impressive zero-shot and few-shot capabilities.
Vision-language models (VLMs) show great promise for 3D scene understanding but are mainly applied to indoor spaces or autonomous driving, focusing on low-level tasks like segmentation. This work expands their use to urban-scale environments by leveraging 3D reconstructions from multi-view aerial imagery. We propose OpenCity3D, an approach that addresses high-level tasks, such as population density estimation, building age classification, property price prediction, crime rate assessment, and noise pollution evaluation. Our findings highlight OpenCity3D's impressive zero-shot and few-shot capabilities, showcasing adaptability to new contexts. This research establishes a new paradigm for language-driven urban analytics, enabling applications in planning, policy, and environmental monitoring. See our project page: opencity3d.github.io