Efficient Motion-Aware Video MLLM
This addresses motion understanding inefficiencies in video MLLMs for applications like video question answering, though it appears incremental as it builds on existing MLLM frameworks with novel input processing.
The paper tackles inefficient data processing and limited motion awareness in video MLLMs by introducing EMA, an Efficient Motion-Aware video MLLM that uses compressed video structures and a motion-aware GOP encoder, achieving state-of-the-art performance on benchmarks like MotionBench and reducing inference costs.
Most current video MLLMs rely on uniform frame sampling and image-level encoders, resulting in inefficient data processing and limited motion awareness. To address these challenges, we introduce EMA, an Efficient Motion-Aware video MLLM that utilizes compressed video structures as inputs. We propose a motion-aware GOP (Group of Pictures) encoder that fuses spatial and motion information within a GOP unit in the compressed video stream, generating compact, informative visual tokens. By integrating fewer but denser RGB frames with more but sparser motion vectors in this native slow-fast input architecture, our approach reduces redundancy and enhances motion representation. Additionally, we introduce MotionBench, a benchmark for evaluating motion understanding across four motion types: linear, curved, rotational, and contact-based. Experimental results show that EMA achieves state-of-the-art performance on both MotionBench and popular video question answering benchmarks, while reducing inference costs. Moreover, EMA demonstrates strong scalability, as evidenced by its competitive performance on long video understanding benchmarks.