Simultaneously Color-Depth Super-Resolution with Conditional Generative Adversarial Network
This addresses the need for improved visual quality in 3D video processing, though it is incremental as it builds on existing GAN frameworks for super-resolution.
The paper tackles the problem of simultaneously enhancing low-resolution color and depth images in 3D videos by proposing a color-depth conditional GAN that leverages mutual information between the two modalities, resulting in high-quality outputs superior to other leading methods.
Recently, Generative Adversarial Network (GAN) has been found wide applications in style transfer, image-to-image translation and image super-resolution. In this paper, a color-depth conditional GAN is proposed to concurrently resolve the problems of depth super-resolution and color super-resolution in 3D videos. Firstly, given the low-resolution depth image and low-resolution color image, a generative network is proposed to leverage mutual information of color image and depth image to enhance each other in consideration of the geometry structural dependency of color-depth image in the same scene. Secondly, three loss functions, including data loss, total variation loss, and 8-connected gradient difference loss are introduced to train this generative network in order to keep generated images close to the real ones, in addition to the adversarial loss. Experimental results demonstrate that the proposed approach produces high-quality color image and depth image from low-quality image pair, and it is superior to several other leading methods. Besides, we use the same neural network framework to resolve the problem of image smoothing and edge detection at the same time.