Toward Accurate and Temporally Consistent Video Restoration from Raw Data
This work addresses video restoration from raw data for applications like computational photography, but it is incremental as it builds on existing VJDD methods with novel losses.
The paper tackled the problem of joint video denoising and demosaicking (VJDD) by proposing a framework with consistent latent space propagation and new losses to improve temporal consistency and accuracy, achieving leading performance in restoration accuracy, perceptual quality, and temporal consistency as demonstrated in extensive experiments.
Denoising and demosaicking are two fundamental steps in reconstructing a clean full-color video from raw data, while performing video denoising and demosaicking jointly, namely VJDD, could lead to better video restoration performance than performing them separately. In addition to restoration accuracy, another key challenge to VJDD lies in the temporal consistency of consecutive frames. This issue exacerbates when perceptual regularization terms are introduced to enhance video perceptual quality. To address these challenges, we present a new VJDD framework by consistent and accurate latent space propagation, which leverages the estimation of previous frames as prior knowledge to ensure consistent recovery of the current frame. A data temporal consistency (DTC) loss and a relational perception consistency (RPC) loss are accordingly designed. Compared with the commonly used flow-based losses, the proposed losses can circumvent the error accumulation problem caused by inaccurate flow estimation and effectively handle intensity changes in videos, improving much the temporal consistency of output videos while preserving texture details. Extensive experiments demonstrate the leading VJDD performance of our method in term of restoration accuracy, perceptual quality and temporal consistency. Codes and dataset are available at \url{https://github.com/GuoShi28/VJDD}.