Perceptual Video Super Resolution with Enhanced Temporal Consistency
This work addresses the challenge of applying perceptual loss functions to video processing for researchers and practitioners in computer vision, representing an incremental advancement over single-image methods.
The paper tackles the problem of flickering artifacts in video super-resolution by introducing a novel adversarial recurrent network that leverages information from previous frames and additional loss functions to enhance temporal consistency, resulting in images with high perceptual quality and improved temporal consistency.
With the advent of perceptual loss functions, new possibilities in super-resolution have emerged, and we currently have models that successfully generate near-photorealistic high-resolution images from their low-resolution observations. Up to now, however, such approaches have been exclusively limited to single image super-resolution. The application of perceptual loss functions on video processing still entails several challenges, mostly related to the lack of temporal consistency of the generated images, i.e., flickering artifacts. In this work, we present a novel adversarial recurrent network for video upscaling that is able to produce realistic textures in a temporally consistent way. The proposed architecture naturally leverages information from previous frames due to its recurrent architecture, i.e. the input to the generator is composed of the low-resolution image and, additionally, the warped output of the network at the previous step. Together with a video discriminator, we also propose additional loss functions to further reinforce temporal consistency in the generated sequences. The experimental validation of our algorithm shows the effectiveness of our approach which obtains images with high perceptual quality and improved temporal consistency.