Hallucination Score: Towards Mitigating Hallucinations in Generative Image Super-Resolution
This addresses a critical but understudied issue in GSR that limits practical deployments by providing a new metric and mitigation strategy for hallucinations.
The paper tackles the problem of hallucinations in generative super-resolution (GSR), where generated details do not match the low-resolution or ground-truth images, by introducing a Hallucination Score (HS) using a multimodal large language model to measure these artifacts and aligning GSR models with deep features to mitigate them, achieving close alignment with human evaluations.
Generative super-resolution (GSR) currently sets the state-of-the-art in terms of perceptual image quality, overcoming the "regression-to-the-mean" blur of prior non-generative models. However, from a human perspective, such models do not fully conform to the optimal balance between quality and fidelity. Instead, a different class of artifacts, in which generated details fail to perceptually match the low resolution image (LRI) or ground-truth image (GTI), is a critical but under studied issue in GSR, limiting its practical deployments. In this work, we focus on measuring, analyzing, and mitigating these artifacts (i.e., "hallucinations"). We observe that hallucinations are not well-characterized with existing image metrics or quality models, as they are orthogonal to both exact fidelity and no-reference quality. Instead, we take advantage of a multimodal large language model (MLLM) by constructing a prompt that assesses hallucinatory visual elements and generates a "Hallucination Score" (HS). We find that our HS is closely aligned with human evaluations, and also provides complementary insights to prior image metrics used for super-resolution (SR) models. In addition, we find certain deep feature distances have strong correlations with HS. We therefore propose to align the GSR models by using such features as differentiable reward functions to mitigate hallucinations.