SceneGram: Conceptualizing and Describing Tangrams in Scene Context
This work addresses the problem of understanding how scene context influences object conceptualization for cognitive science and AI researchers, but it is incremental as it builds on existing datasets and analyses.
The paper introduces SceneGram, a dataset of human references to tangram shapes in various scene contexts, enabling analysis of how scene context affects conceptualization, and finds that multimodal LLMs fail to capture the richness and variability of human conceptualizations.
Research on reference and naming suggests that humans can come up with very different ways of conceptualizing and referring to the same object, e.g. the same abstract tangram shape can be a "crab", "sink" or "space ship". Another common assumption in cognitive science is that scene context fundamentally shapes our visual perception of objects and conceptual expectations. This paper contributes SceneGram, a dataset of human references to tangram shapes placed in different scene contexts, allowing for systematic analyses of the effect of scene context on conceptualization. Based on this data, we analyze references to tangram shapes generated by multimodal LLMs, showing that these models do not account for the richness and variability of conceptualizations found in human references.