M3-CVC: Controllable Video Compression with Multimodal Generative Models
This addresses the challenge of efficient and interpretable video compression for applications requiring ultra-low bitrates, representing a novel method rather than an incremental improvement.
The paper tackles the problem of limited controllability and generality in video compression at ultra-low bitrates by proposing M3-CVC, a framework that uses multimodal generative models, achieving significant performance improvements over the state-of-the-art VVC standard in preserving semantic and perceptual fidelity.
Traditional and neural video codecs commonly encounter limitations in controllability and generality under ultra-low-bitrate coding scenarios. To overcome these challenges, we propose M3-CVC, a controllable video compression framework incorporating multimodal generative models. The framework utilizes a semantic-motion composite strategy for keyframe selection to retain critical information. For each keyframe and its corresponding video clip, a dialogue-based large multimodal model (LMM) approach extracts hierarchical spatiotemporal details, enabling both inter-frame and intra-frame representations for improved video fidelity while enhancing encoding interpretability. M3-CVC further employs a conditional diffusion-based, text-guided keyframe compression method, achieving high fidelity in frame reconstruction. During decoding, textual descriptions derived from LMMs guide the diffusion process to restore the original video's content accurately. Experimental results demonstrate that M3-CVC significantly outperforms the state-of-the-art VVC standard in ultra-low bitrate scenarios, particularly in preserving semantic and perceptual fidelity.