Understanding Degradation with Vision Language Model
This work addresses the challenge of parametric physics understanding in image degradations for computer vision applications, representing a novel method rather than an incremental improvement.
The paper tackles the problem of understanding visual degradations by redefining it as a hierarchical structured prediction task and introduces DU-VLM, a multimodal chain-of-thought model that unifies degradation type, parameter key, and continuous value estimation under an autoregressive framework. The approach significantly outperforms generalist baselines in accuracy and robustness, generalizes to unseen distributions, and enables zero-shot control of diffusion models for image restoration without fine-tuning.
Understanding visual degradations is a critical yet challenging problem in computer vision. While recent Vision-Language Models (VLMs) excel at qualitative description, they often fall short in understanding the parametric physics underlying image degradations. In this work, we redefine degradation understanding as a hierarchical structured prediction task, necessitating the concurrent estimation of degradation types, parameter keys, and their continuous physical values. Although these sub-tasks operate in disparate spaces, we prove that they can be unified under one autoregressive next-token prediction paradigm, whose error is bounded by the value-space quantization grid. Building on this insight, we introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning using structured rewards. Furthermore, we show that DU-VLM can serve as a zero-shot controller for pre-trained diffusion models, enabling high-fidelity image restoration without fine-tuning the generative backbone. We also introduce \textbf{DU-110k}, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations. Extensive experiments demonstrate that our approach significantly outperforms generalist baselines in both accuracy and robustness, exhibiting generalization to unseen distributions.