Perceive, Understand and Restore: Real-World Image Super-Resolution with Autoregressive Multimodal Generative Models
This work addresses the challenge of robust super-resolution for real-world images, particularly in complex scenarios, representing an incremental improvement over existing methods that use text-to-image diffusion models.
The paper tackles the problem of inaccurate and unnatural reconstructions in real-world image super-resolution (Real-ISR) for complex or heavily degraded scenes by proposing PURE, a method that adapts a pre-trained autoregressive multimodal model to perceive, understand, and restore images, resulting in preserved content and realistic details in complex scenes.
By leveraging the generative priors from pre-trained text-to-image diffusion models, significant progress has been made in real-world image super-resolution (Real-ISR). However, these methods tend to generate inaccurate and unnatural reconstructions in complex and/or heavily degraded scenes, primarily due to their limited perception and understanding capability of the input low-quality image. To address these limitations, we propose, for the first time to our knowledge, to adapt the pre-trained autoregressive multimodal model such as Lumina-mGPT into a robust Real-ISR model, namely PURE, which Perceives and Understands the input low-quality image, then REstores its high-quality counterpart. Specifically, we implement instruction tuning on Lumina-mGPT to perceive the image degradation level and the relationships between previously generated image tokens and the next token, understand the image content by generating image semantic descriptions, and consequently restore the image by generating high-quality image tokens autoregressively with the collected information. In addition, we reveal that the image token entropy reflects the image structure and present a entropy-based Top-k sampling strategy to optimize the local structure of the image during inference. Experimental results demonstrate that PURE preserves image content while generating realistic details, especially in complex scenes with multiple objects, showcasing the potential of autoregressive multimodal generative models for robust Real-ISR. The model and code will be available at https://github.com/nonwhy/PURE.