Eyes on Target: Gaze-Aware Object Detection in Egocentric Video
This work addresses improved object detection for egocentric video analysis, particularly in simulation scenarios for human performance evaluation, but it is incremental as it builds on existing gaze and transformer methods.
The paper tackles object detection in egocentric videos by integrating human gaze cues into a Vision Transformer to bias attention toward attended regions, resulting in consistent accuracy gains over gaze-agnostic baselines on simulator and public datasets.
Human gaze offers rich supervisory signals for understanding visual attention in complex visual environments. In this paper, we propose Eyes on Target, a novel depth-aware and gaze-guided object detection framework designed for egocentric videos. Our approach injects gaze-derived features into the attention mechanism of a Vision Transformer (ViT), effectively biasing spatial feature selection toward human-attended regions. Unlike traditional object detectors that treat all regions equally, our method emphasises viewer-prioritised areas to enhance object detection. We validate our method on an egocentric simulator dataset where human visual attention is critical for task assessment, illustrating its potential in evaluating human performance in simulation scenarios. We evaluate the effectiveness of our gaze-integrated model through extensive experiments and ablation studies, demonstrating consistent gains in detection accuracy over gaze-agnostic baselines on both the custom simulator dataset and public benchmarks, including Ego4D Ego-Motion and Ego-CH-Gaze datasets. To interpret model behaviour, we also introduce a gaze-aware attention head importance metric, revealing how gaze cues modulate transformer attention dynamics.