GazeLLM: Multimodal LLMs incorporating Human Visual Attention
This provides an efficient solution for human-computer interaction and augmentation applications, such as skill transfer, by enabling MLLMs to handle longer and higher-resolution videos without compromising performance.
The paper tackles the problem of high memory and processing demands in Multimodal LLMs (MLLMs) when analyzing high-resolution, long-duration first-person videos by integrating eye-tracking data to selectively process gazed-focused regions, achieving equivalent or better task comprehension with a tenfold reduction in video data input.
Large Language Models (LLMs) are advancing into Multimodal LLMs (MLLMs), capable of processing image, audio, and video as well as text. Combining first-person video, MLLMs show promising potential for understanding human activities through video and audio, enabling many human-computer interaction and human-augmentation applications such as human activity support, real-world agents, and skill transfer to robots or other individuals. However, handling high-resolution, long-duration videos generates large latent representations, leading to substantial memory and processing demands, limiting the length and resolution MLLMs can manage. Reducing video resolution can lower memory usage but often compromises comprehension. This paper introduces a method that optimizes first-person video analysis by integrating eye-tracking data, and proposes a method that decomposes first-person vision video into sub areas for regions of gaze focus. By processing these selectively gazed-focused inputs, our approach achieves task comprehension equivalent to or even better than processing the entire image at full resolution, but with significantly reduced video data input (reduce the number of pixels to one-tenth), offering an efficient solution for using MLLMs to interpret and utilize human skills.