Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated
It addresses the need for reliability-aware benchmarking in urban perception for governance stakeholders, but the argument is primarily conceptual with a small-scale illustrative study.
The paper argues that benchmarks for vision-language models in urban perception should treat annotator disagreement and abstention as measurement outcomes, and reports that model agreement with human consensus co-varies with human reliability across 30 dimensions in a Montreal street scene dataset.
Vision-language models (VLMs) are increasingly used to generate structured descriptions of street-level imagery for tasks such as streetscape auditing, mapping, and public consultation. These uses combine observable attributes with appraisal categories, and the human targets are often distributions of judgments with disagreement and explicit non-response. This paper argues that benchmarking VLMs for urban perception should treat disagreement and abstention as measurement outcomes, report inter-annotator reliability alongside model alignment, and treat the label space and scoring policy as negotiable artifacts when outputs are intended to inform urban governance. We ground the argument in a benchmark of 100 Montreal street scenes annotated along 30 dimensions by 12 participants from seven community organizations, and in a deterministic zero-shot evaluation of seven VLMs. Across dimensions, model agreement with human consensus co-varies with dimension-level human reliability, and for the appraisal dimension Overall Impression models and annotators exhibit distributional mismatch including different rates of Not applicable. We close with actions for benchmark creators, model developers, and institutions to make uncertainty and benchmark assumptions visible in evaluation reports.