ELSA: Evaluating Localization of Social Activities in Urban Streets using Open-Vocabulary Detection
This work addresses the lack of benchmarks for evaluating social activity localization in computer vision, which is important for urban analysis and AI applications, but it is incremental as it builds on existing OVD methods.
The authors tackled the problem of inconsistent performance and poor calibration in Open Vocabulary Detection models for social activities in urban streets by introducing ELSA, a new multi-label benchmark with over 900 annotated images and 4,300 bounding boxes, and reported results showing limitations in existing SOTA models like Grounding DINO and OWL.
Existing Open Vocabulary Detection (OVD) models exhibit a number of challenges. They often struggle with semantic consistency across diverse inputs, and are often sensitive to slight variations in input phrasing, leading to inconsistent performance. The calibration of their predictive confidence, especially in complex multi-label scenarios, remains suboptimal, frequently resulting in overconfident predictions that do not accurately reflect their context understanding. To understand these limitations, multi-label detection benchmarks are needed. A particularly challenging domain for such benchmarking is social activities. Due to the lack of multi-label benchmarks for social interactions, in this work we present ELSA: Evaluating Localization of Social Activities. ELSA draws on theoretical frameworks in urban sociology and design and uses in-the-wild street-level imagery, where the size of groups and the types of activities vary significantly. ELSA includes more than 900 manually annotated images with more than 4,300 multi-labeled bounding boxes for individual and group activities. We introduce a novel confidence score computation method NLSE and a novel Dynamic Box Aggregation (DBA) algorithm to assess semantic consistency in overlapping predictions. We report our results on the widely-used SOTA models Grounding DINO, Detic, OWL, and MDETR. Our evaluation protocol considers semantic stability and localization accuracy and further exposes the limitations of existing approaches.