GazeMoE: Perception of Gaze Target with Mixture-of-Experts
This work addresses the challenge of generalizable gaze target estimation for robots, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the problem of estimating human gaze targets from images for robot understanding of human attention, proposing GazeMoE, a framework that uses Mixture-of-Experts modules to integrate multi-modal cues from a frozen foundation model, and it achieves state-of-the-art performance on benchmark datasets.
Estimating human gaze target from visible images is a critical task for robots to understand human attention, yet the development of generalizable neural architectures and training paradigms remains challenging. While recent advances in pre-trained vision foundation models offer promising avenues for locating gaze targets, the integration of multi-modal cues -- including eyes, head poses, gestures, and contextual features -- demands adaptive and efficient decoding mechanisms. Inspired by Mixture-of-Experts (MoE) for adaptive domain expertise in large vision-language models, we propose GazeMoE, a novel end-to-end framework that selectively leverages gaze-target-related cues from a frozen foundation model through MoE modules. To address class imbalance in gaze target classification (in-frame vs. out-of-frame) and enhance robustness, GazeMoE incorporates a class-balancing auxiliary loss alongside strategic data augmentations, including region-specific cropping and photometric transformations. Extensive experiments on benchmark datasets demonstrate that our GazeMoE achieves state-of-the-art performance, outperforming existing methods on challenging gaze estimation tasks. The code and pre-trained models are released at https://huggingface.co/zdai257/GazeMoE