When Video Coding Meets Multimodal Large Language Models: A Unified Paradigm for Video Coding
This work addresses video coding efficiency by integrating MLLMs, offering a novel approach that could impact multimedia and AI applications, though it appears incremental in building on existing generative models.
The paper tackles video compression by introducing a unified paradigm called Cross-Modality Video Coding (CMVC) that leverages Multimodal Large Language Models (MLLMs) to disentangle videos into spatial content and motion components for compact representation, with results showing effective semantic reconstruction and competitive perceptual consistency in experiments.
Existing codecs are designed to eliminate intrinsic redundancies to create a compact representation for compression. However, strong external priors from Multimodal Large Language Models (MLLMs) have not been explicitly explored in video compression. Herein, we introduce a unified paradigm for Cross-Modality Video Coding (CMVC), which is a pioneering approach to explore multimodality representation and video generative models in video coding. Specifically, on the encoder side, we disentangle a video into spatial content and motion components, which are subsequently transformed into distinct modalities to achieve very compact representation by leveraging MLLMs. During decoding, previously encoded components and video generation models are leveraged to create multiple encoding-decoding modes that optimize video reconstruction quality for specific decoding requirements, including Text-Text-to-Video (TT2V) mode to ensure high-quality semantic information and Image-Text-to-Video (IT2V) mode to achieve superb perceptual consistency. In addition, we propose an efficient frame interpolation model for IT2V mode via Low-Rank Adaption (LoRA) tuning to guarantee perceptual quality, which allows the generated motion cues to behave smoothly. Experiments on benchmarks indicate that TT2V achieves effective semantic reconstruction, while IT2V exhibits competitive perceptual consistency. These results highlight potential directions for future research in video coding.