Do Vision-Language Models See Urban Scenes as People Do? An Urban Perception Benchmark
This work addresses the need for reproducible evaluation of VLMs in urban perception, which is important for participatory urban analysis, but it is incremental as it introduces a small benchmark with limited data.
The authors tackled the problem of evaluating how well vision-language models (VLMs) understand urban scenes compared to humans by introducing a benchmark with 100 Montreal street images and human annotations across 30 dimensions. The results showed that VLMs perform better on objective properties than subjective appraisals, with the top model achieving a macro score of 0.31 and mean Jaccard of 0.48 on multi-label items.
Understanding how people read city scenes can inform design and planning. We introduce a small benchmark for testing vision-language models (VLMs) on urban perception using 100 Montreal street images, evenly split between photographs and photorealistic synthetic scenes. Twelve participants from seven community groups supplied 230 annotation forms across 30 dimensions mixing physical attributes and subjective impressions. French responses were normalized to English. We evaluated seven VLMs in a zero-shot setup with a structured prompt and deterministic parser. We use accuracy for single-choice items and Jaccard overlap for multi-label items; human agreement uses Krippendorff's alpha and pairwise Jaccard. Results suggest stronger model alignment on visible, objective properties than subjective appraisals. The top system (claude-sonnet) reaches macro 0.31 and mean Jaccard 0.48 on multi-label items. Higher human agreement coincides with better model scores. Synthetic images slightly lower scores. We release the benchmark, prompts, and harness for reproducible, uncertainty-aware evaluation in participatory urban analysis.