Defining and Evaluating Visual Language Models' Basic Spatial Abilities: A Perspective from Psychometrics
This provides a diagnostic toolkit for spatial intelligence evaluation in VLMs, with implications for embodied AI development, but it is incremental as it applies existing psychometric methods to a new domain.
The paper tackled the problem of evaluating spatial abilities in Visual Language Models (VLMs) by defining five Basic Spatial Abilities (BSAs) and benchmarking 13 models, revealing significant gaps versus humans (average score 24.95 vs. 68.38) and identifying key findings such as model hierarchies and intervention limits.
The Theory of Multiple Intelligences underscores the hierarchical nature of cognitive capabilities. To advance Spatial Artificial Intelligence, we pioneer a psychometric framework defining five Basic Spatial Abilities (BSAs) in Visual Language Models (VLMs): Spatial Perception, Spatial Relation, Spatial Orientation, Mental Rotation, and Spatial Visualization. Benchmarking 13 mainstream VLMs through nine validated psychometric experiments reveals significant gaps versus humans (average score 24.95 vs. 68.38), with three key findings: 1) VLMs mirror human hierarchies (strongest in 2D orientation, weakest in 3D rotation) with independent BSAs (Pearson's r<0.4); 2) Smaller models such as Qwen2-VL-7B surpass larger counterparts, with Qwen leading (30.82) and InternVL2 lagging (19.6); 3) Interventions like chain-of-thought (0.100 accuracy gain) and 5-shot training (0.259 improvement) show limits from architectural constraints. Identified barriers include weak geometry encoding and missing dynamic simulation. By linking psychometric BSAs to VLM capabilities, we provide a diagnostic toolkit for spatial intelligence evaluation, methodological foundations for embodied AI development, and a cognitive science-informed roadmap for achieving human-like spatial intelligence.