Human Perceptual Evaluations for Image Compression
This addresses a critical evaluation issue for researchers and practitioners in image compression, highlighting a misleading metric that could affect algorithm development and comparisons.
The paper tackles the problem of evaluating image compression schemes by showing that deep learning techniques optimized for MS-SSIM can have higher scores but be perceptually worse than engineered methods with lower scores, as demonstrated through user studies.
Recently, there has been much interest in deep learning techniques to do image compression and there have been claims that several of these produce better results than engineered compression schemes (such as JPEG, JPEG2000 or BPG). A standard way of comparing image compression schemes today is to use perceptual similarity metrics such as PSNR or MS-SSIM (multi-scale structural similarity). This has led to some deep learning techniques which directly optimize for MS-SSIM by choosing it as a loss function. While this leads to a higher MS-SSIM for such techniques, we demonstrate using user studies that the resulting improvement may be misleading. Deep learning techniques for image compression with a higher MS-SSIM may actually be perceptually worse than engineered compression schemes with a lower MS-SSIM.