Maintaining Natural Image Statistics with the Contextual Loss
This addresses the challenge of photorealism in image restoration and generation for computer vision applications, offering a complementary approach to GANs.
The paper tackles the problem of generating realistic images by training CNNs to maintain natural image statistics, reducing training data needs by orders of magnitude and achieving state-of-the-art results in super-resolution and surface normal estimation.
Maintaining natural image statistics is a crucial factor in restoration and generation of realistic looking images. When training CNNs, photorealism is usually attempted by adversarial training (GAN), that pushes the output images to lie on the manifold of natural images. GANs are very powerful, but not perfect. They are hard to train and the results still often suffer from artifacts. In this paper we propose a complementary approach, that could be applied with or without GAN, whose goal is to train a feed-forward CNN to maintain natural internal statistics. We look explicitly at the distribution of features in an image and train the network to generate images with natural feature distributions. Our approach reduces by orders of magnitude the number of images required for training and achieves state-of-the-art results on both single-image super-resolution, and high-resolution surface normal estimation.