Pixels, Patterns, but No Poetry: To See The World like Humans
This addresses a key gap in AI for developing more human-like perception in MLLMs, though it is incremental as it focuses on benchmarking rather than solving the perception problem directly.
The paper tackles the problem of whether Multimodal Large Language Models (MLLMs) can perceive the world like humans by introducing the Turing Eye Test (TET), a perception-oriented benchmark with synthetic images, and finds that state-of-the-art MLLMs fail catastrophically on tasks trivial for humans, with fine-tuning the vision tower enabling rapid adaptation.
Achieving human-like perception and reasoning in Multimodal Large Language Models (MLLMs) remains a central challenge in artificial intelligence. While recent research has primarily focused on enhancing reasoning capabilities in MLLMs, a fundamental question persists: Can Multimodal Large Language Models truly perceive the world as humans do? This paper shifts focus from reasoning to perception. Rather than constructing benchmarks specifically for reasoning, we introduce the Turing Eye Test (TET), a challenging perception-oriented benchmark comprising four diagnostic tasks that evaluate MLLMs' performance on synthetic images that humans process intuitively. Our findings reveal that state-of-the-art MLLMs exhibit catastrophic failures on our perceptual tasks trivial for humans. Both in-context learning and training on language backbone-effective for previous benchmarks-fail to improve performance on our tasks, while fine-tuning the vision tower enables rapid adaptation, suggesting that our benchmark poses challenges for vision tower generalization rather than for the knowledge and reasoning capabilities of the language backbone-a key gap between current MLLMs and human perception. We release a representative subset of TET tasks in this version, and will introduce more diverse tasks and methods to enhance visual generalization in future work.