EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenes
For researchers evaluating MLLMs' egocentric reasoning, this benchmark reveals a critical gap in current models' ability to select actions from their own perspective, highlighting the need for targeted training beyond scene understanding.
EgoGapBench is a diagnostic benchmark isolating egocentric perspective understanding for action selection in multi-agent scenes. Humans perform reliably, while MLLMs (both open-source and proprietary) perform substantially worse, systematically selecting actions of other visible agents; fine-tuning on existing egocentric data fails to close the gap.
Existing egocentric benchmarks have primarily constructed the egocentric setting from first-person-view data, which makes it difficult to evaluate egocentric perspective itself in isolation. However, understanding first-person-view input and taking an egocentric perspective are separable abilities, especially when first-person body cues are absent or when other agents are present. To isolate egocentric perspective understanding, we introduce EgoGapBench, a diagnostic benchmark for measuring action selection in multi-agent egocentric scenes. We define the ability measured by this benchmark as Egocentric Action Selection (EAS): selecting an appropriate action from the agent's perspective in the presence of other agents. On EgoGapBench, humans answer reliably, whereas both open-source and proprietary MLLMs perform substantially worse and systematically select actions performed by other visible agents. Fine-tuning on existing egocentric data fails to close this gap and can even be detrimental. In contrast, fine-tuning on EgoGapBench training data improves accuracy but does not reach human performance. These results show that EAS is difficult to acquire from first-person-view data alone, and that MLLMs should be evaluated and trained not only for scene understanding but also for egocentric action selection.