EgoAVU: Egocentric Audio-Visual Understanding
This work addresses the problem of MLLMs failing to jointly understand audio and visual cues in egocentric videos, which is crucial for embodied intelligence, by providing a scalable data generation and evaluation framework. This is an incremental improvement on existing MLLMs.
This paper introduces EgoAVU, a data engine that automatically generates egocentric audio-visual narrations, questions, and answers to address the lack of joint-modality understanding in MLLMs for egocentric videos. By finetuning MLLMs on the resulting 3M-sample EgoAVU-Instruct dataset, the authors achieve up to 113% performance improvement on their EgoAVU-Bench and up to 28% relative gain on other benchmarks like EgoTempo and EgoIllusion.
Understanding egocentric videos plays a vital role for embodied intelligence. Recent multi-modal large language models (MLLMs) can accept both visual and audio inputs. However, due to the challenge of obtaining text labels with coherent joint-modality information, whether MLLMs can jointly understand both modalities in egocentric videos remains under-explored. To address this problem, we introduce EgoAVU, a scalable data engine to automatically generate egocentric audio-visual narrations, questions, and answers. EgoAVU enriches human narrations with multimodal context and generates audio-visual narrations through cross-modal correlation modeling. Token-based video filtering and modular, graph-based curation ensure both data diversity and quality. Leveraging EgoAVU, we construct EgoAVU-Instruct, a large-scale training dataset of 3M samples, and EgoAVU-Bench, a manually verified evaluation split covering diverse tasks. EgoAVU-Bench clearly reveals the limitations of existing MLLMs: they bias heavily toward visual signals, often neglecting audio cues or failing to correspond audio with the visual source. Finetuning MLLMs on EgoAVU-Instruct effectively addresses this issue, enabling up to 113% performance improvement on EgoAVU-Bench. Such benefits also transfer to other benchmarks such as EgoTempo and EgoIllusion, achieving up to 28% relative performance gain. Code will be released to the community.