Failures in Perspective-taking of Multimodal AI Systems
This work identifies a key limitation in AI spatial cognition for researchers and developers, though it is incremental as it extends prior research on multimodal systems.
The study assessed GPT-4o's perspective-taking abilities using cognitive science techniques, revealing that multimodal AI systems rely on propositional representations unlike human analog cognition, which limits their spatial understanding.
This study extends previous research on spatial representations in multimodal AI systems. Although current models demonstrate a rich understanding of spatial information from images, this information is rooted in propositional representations, which differ from the analog representations employed in human and animal spatial cognition. To further explore these limitations, we apply techniques from cognitive and developmental science to assess the perspective-taking abilities of GPT-4o. Our analysis enables a comparison between the cognitive development of the human brain and that of multimodal AI, offering guidance for future research and model development.