XPSR: Cross-modal Priors for Diffusion-based Image Super-Resolution
This work addresses image quality restoration for applications like photography or surveillance, but it is incremental as it builds on existing diffusion-based methods with multimodal enhancements.
The paper tackles the challenge of image super-resolution (ISR) where low-resolution images suffer from severe degradation, leading to incorrect content or artifacts, by proposing the XPSR framework that uses cross-modal priors from MLLMs to generate high-fidelity and high-realism images across synthetic and real-world datasets.
Diffusion-based methods, endowed with a formidable generative prior, have received increasing attention in Image Super-Resolution (ISR) recently. However, as low-resolution (LR) images often undergo severe degradation, it is challenging for ISR models to perceive the semantic and degradation information, resulting in restoration images with incorrect content or unrealistic artifacts. To address these issues, we propose a \textit{Cross-modal Priors for Super-Resolution (XPSR)} framework. Within XPSR, to acquire precise and comprehensive semantic conditions for the diffusion model, cutting-edge Multimodal Large Language Models (MLLMs) are utilized. To facilitate better fusion of cross-modal priors, a \textit{Semantic-Fusion Attention} is raised. To distill semantic-preserved information instead of undesired degradations, a \textit{Degradation-Free Constraint} is attached between LR and its high-resolution (HR) counterpart. Quantitative and qualitative results show that XPSR is capable of generating high-fidelity and high-realism images across synthetic and real-world datasets. Codes are released at \url{https://github.com/qyp2000/XPSR}.