Can Vision Language Models Infer Human Gaze Direction? A Controlled Study
This work addresses the problem of enabling natural human-AI interaction by assessing VLMs' gaze inference abilities, revealing significant limitations compared to humans, which is incremental as it benchmarks existing models without introducing new methods.
The study evaluated 111 Vision Language Models (VLMs) and 65 humans on inferring gaze direction from photos, finding that 94 VLMs performed no better than random guessing, while humans achieved near-ceiling accuracy, with top-tier VLMs showing above-chance but imperfect performance that declined with task difficulty.
The ability to infer what others are looking at is a critical component of a theory of mind that underpins natural human-AI interaction. We characterized this skill in 111 Vision Language Models (VLMs) and human participants (N = 65) using photos taken with manipulated difficulty and variability. We found that 94 of the 111 VLMs were not better than random guessing, while humans achieved near-ceiling accuracy. VLMs respond with each choice almost equally frequently. Are they randomly guessing? At least for five top-tier VLMs, their performance was above chance, declined with increasing task difficulty, but barely varied across different prompts and scene objects. These behavioral patterns cannot be explained by considering VLMs as random guessers. Instead, they likely utilize head orientation but not eye appearance to infer gaze direction, such that their performance is imperfect, subject to the task difficulty, but robust to superficial perceptual variations. This suggests that VLMs, lacking effective gaze inference skills, have yet to become technologies that can naturally interact with humans, but the potential remains.