UniMMVSR: A Unified Multi-Modal Framework for Cascaded Video Super-Resolution
This addresses the limitation of existing text-only methods in multi-modal video generation for applications requiring high-fidelity video production.
The paper tackles the problem of cascaded video super-resolution by introducing UniMMVSR, a unified framework that incorporates hybrid-modal conditions like text, images, and videos, resulting in significantly outperforming existing methods with superior detail and conformity to conditions, including enabling 4K video generation.
Cascaded video super-resolution has emerged as a promising technique for decoupling the computational burden associated with generating high-resolution videos using large foundation models. Existing studies, however, are largely confined to text-to-video tasks and fail to leverage additional generative conditions beyond text, which are crucial for ensuring fidelity in multi-modal video generation. We address this limitation by presenting UniMMVSR, the first unified generative video super-resolution framework to incorporate hybrid-modal conditions, including text, images, and videos. We conduct a comprehensive exploration of condition injection strategies, training schemes, and data mixture techniques within a latent video diffusion model. A key challenge was designing distinct data construction and condition utilization methods to enable the model to precisely utilize all condition types, given their varied correlations with the target video. Our experiments demonstrate that UniMMVSR significantly outperforms existing methods, producing videos with superior detail and a higher degree of conformity to multi-modal conditions. We also validate the feasibility of combining UniMMVSR with a base model to achieve multi-modal guided generation of 4K video, a feat previously unattainable with existing techniques.