Beyond Fidelity: Semantic Similarity Assessment in Low-Level Image Processing
For researchers in low-level image processing, it addresses the need for semantic-level evaluation beyond visual fidelity, which is increasingly important with generative models.
The paper formalizes Semantic Similarity as a new evaluation task for low-level image processing and proposes T3S, which models image semantics via foreground entities, background entities, and relations. T3S outperforms existing metrics on COCO and SPA-Data, better reflecting semantic changes under degradations.
Low-level image processing has long been evaluated mainly from the perspective of visual fidelity. However, with the rise of deep learning and generative models, processed images may preserve perceptual quality while altering semantic content, making conventional Image Quality Assessment (IQA) insufficient for semantic-level assessment. In this paper, we formalize \textit{Semantic Similarity} as a new evaluation task for low-level image processing, aimed at measuring whether semantic content is preserved after processing. We further present a structured formulation of image semantics based on semantic entities and their relations, and discuss the desired properties and constraints of a valid semantic similarity index. Based on this formulation, we propose Triplet-based Semantic Similarity Score (T3S), which models image semantics through foreground entities, background entities, and relations. T3S combines semantic entity extraction, foreground-background disentanglement, and open-world class/relation modeling. Experiments on COCO and SPA-Data show that T3S consistently outperforms existing fidelity-oriented metrics and representative semantic-level baselines, while better reflecting progressive semantic changes under diverse degradations. These results highlight the importance of semantic assessment in modern low-level vision.