Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding
This work addresses the challenge of improving egocentric VLMs for applications like assistive robots and human-machine collaboration, though it is incremental as it builds on existing VLM architectures.
The paper tackles the problem of enhancing vision-language models (VLMs) for egocentric understanding by using gaze data during training to align model attention with human visual focus, resulting in improvements of up to 11% in future event prediction and around 7% in current activity understanding compared to baselines.
Eye gaze offers valuable cues about attention, short-term intent, and future actions, making it a powerful signal for modeling egocentric behavior. In this work, we propose a gaze-regularized framework that enhances VLMs for two key egocentric understanding tasks: fine-grained future event prediction and current activity understanding. Unlike prior approaches that rely solely on visual inputs or use gaze as an auxiliary input signal , our method uses gaze only during training. We introduce a gaze-regularized attention mechanism that aligns model focus with human visual gaze. This design is flexible and modular, allowing it to generalize across multiple VLM architectures that utilize attention. Experimental results show that our approach improves semantic prediction scores by up to 11 for future event prediction and around 7 for current activity understanding, compared to the corresponding baseline models trained without gaze regularization. These results highlight the value of gaze-guided training in improving the accuracy and robustness of egocentric VLMs. Overall, this work establishes a foundation for using human gaze to enhance the predictive capabilities of VLMs in real-world scenarios like assistive robots and human-machine collaboration. Code and additional information is available at: https://github.com/anupampani/Gaze-VLM