DisCo: Towards Distinct and Coherent Visual Encapsulation in Video MLLMs
This work addresses a specific bottleneck in video MLLMs for researchers and practitioners, offering an incremental improvement over existing encapsulation methods.
The paper tackles the problem of semantic indistinctness and temporal incoherence in visual encapsulation for video Multimodal Large Language Models (video MLLMs) by introducing DisCo, a method that integrates a Visual Concept Discriminator and Temporal Focus Calibrator, resulting in outperforming previous state-of-the-art methods across various video understanding benchmarks with higher token efficiency.
In video Multimodal Large Language Models (video MLLMs), the visual encapsulation process plays a pivotal role in converting video contents into representative tokens for LLM input. While linear projectors are widely employed for encapsulation, they introduce semantic indistinctness and temporal incoherence when applied to videos. Conversely, the structure of resamplers shows promise in tackling these challenges, but an effective solution remains unexplored. Drawing inspiration from resampler structures, we introduce DisCo, a novel visual encapsulation method designed to yield semantically distinct and temporally coherent visual tokens for video MLLMs. DisCo integrates two key components: (1) A Visual Concept Discriminator (VCD) module, assigning unique semantics for visual tokens by associating them in pair with discriminative concepts in the video. (2) A Temporal Focus Calibrator (TFC) module, ensuring consistent temporal focus of visual tokens to video elements across every video frame. Through extensive experiments on multiple video MLLM frameworks, we demonstrate that DisCo remarkably outperforms previous state-of-the-art methods across a variety of video understanding benchmarks, while also achieving higher token efficiency thanks to the reduction of semantic indistinctness. The code: https://github.com/ZJHTerry18/DisCo.