Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attention
This work addresses the challenge of fine-grained activity description in egocentric vision for applications like assistive technology or human-computer interaction, though it is incremental as it builds on existing video captioning with sensor augmentation.
The paper tackles the problem of generating detailed descriptions of human activities from egocentric video by introducing a sensor-augmented captioning task, a new dataset (MMAC Captions), and a method that fuses video and motion sensor data using dynamic modal attention, outperforming strong baselines on the dataset.
Automatically describing video, or video captioning, has been widely studied in the multimedia field. This paper proposes a new task of sensor-augmented egocentric-video captioning, a newly constructed dataset for it called MMAC Captions, and a method for the newly proposed task that effectively utilizes multi-modal data of video and motion sensors, or inertial measurement units (IMUs). While conventional video captioning tasks have difficulty in dealing with detailed descriptions of human activities due to the limited view of a fixed camera, egocentric vision has greater potential to be used for generating the finer-grained descriptions of human activities on the basis of a much closer view. In addition, we utilize wearable-sensor data as auxiliary information to mitigate the inherent problems in egocentric vision: motion blur, self-occlusion, and out-of-camera-range activities. We propose a method for effectively utilizing the sensor data in combination with the video data on the basis of an attention mechanism that dynamically determines the modality that requires more attention, taking the contextual information into account. We compared the proposed sensor-fusion method with strong baselines on the MMAC Captions dataset and found that using sensor data as supplementary information to the egocentric-video data was beneficial, and that our proposed method outperformed the strong baselines, demonstrating the effectiveness of the proposed method.