Do image and video quality metrics model low-level human vision?
This work provides a new evaluation framework for quality metrics, addressing a gap in assessing their perceptual alignment for researchers in computer vision and multimedia.
The paper tackles the problem of evaluating whether image and video quality metrics accurately model low-level human vision by proposing tests for contrast sensitivity, masking, and matching, and finds that metrics like LPIPS and MS-SSIM perform well in some aspects while VMAF and SSIM have weaknesses.
Image and video quality metrics, such as SSIM, LPIPS, and VMAF, are aimed to predict the perceived quality of the evaluated content and are often claimed to be "perceptual". Yet, few metrics directly model human visual perception, and most rely on hand-crafted formulas or training datasets to achieve alignment with perceptual data. In this paper, we propose a set of tests for full-reference quality metrics that examine their ability to model several aspects of low-level human vision: contrast sensitivity, contrast masking, and contrast matching. The tests are meant to provide additional scrutiny for newly proposed metrics. We use our tests to analyze 33 existing image and video quality metrics and find their strengths and weaknesses, such as the ability of LPIPS and MS-SSIM to predict contrast masking and poor performance of VMAF in this task. We further find that the popular SSIM metric overemphasizes differences in high spatial frequencies, but its multi-scale counterpart, MS-SSIM, addresses this shortcoming. Such findings cannot be easily made using existing evaluation protocols.