MAPS: A Synthetic Dataset for Probing Vision Models in a Controlled 3D Scene Space
For researchers studying vision model robustness, MAPS provides a controlled instrument to attribute model behavior to specific scene parameters, revealing that camera distance and elevation are universal failure axes.
MAPS introduces a synthetic dataset with 2,618 photorealistic 3D meshes and a rendering pipeline to control nine scene factors (background, camera, lighting). Evaluating 20 vision models, they find that camera distance and elevation dominate recognition failures across all architectures, and that modern CNNs and transformers cluster together in sensitivity profiles, distinct from older architectures.
Modern vision models achieve strong performance on standard benchmarks, yet their aggregate accuracy reveals little about which scene properties drive their predictions. Existing robustness benchmarks provide important stress tests, but typically manipulate global 2D image properties, rely on entangled real-world variation, or cover only a limited set of 3D objects and scene parameters. We introduce MAPS (Manifolds of Artificial Parametric Scenes), a scalable instrument for controlled attribution of vision model behavior to scene parameters. MAPS comprises 2,618 curated photorealistic 3D meshes validated for recognizability across 560 ImageNet classes and provides a Blender-based rendering pipeline for on-demand image generation under continuous variation of nine independent scene-factors spanning background, camera, and lighting, extensible to other factors. To showcase its applicability, we use MAPS to evaluate 20 convolutional and transformer-based models by quantifying their reliance on these scene factors through regression-based sensitivity analysis. We find a near-universal failure axis across all tested architectures: camera distance and elevation consistently dominate recognition failure regardless of ImageNet accuracy. However, the full sensitivity structure reveals that modern CNNs and transformers cluster together, distinct from older architectures, suggesting that fine-grained architectural design choices, rather than the coarse CNN-versus-transformer distinction, are the stronger determinant of sensitivity profiles.