Progressive Training of A Two-Stage Framework for Video Restoration
This work addresses training difficulties in video restoration models, offering practical improvements for compressed video enhancement and super-resolution, though it appears incremental as it builds on existing methods.
The authors tackled the problem of training recurrent and transformer models for video restoration, which is costly and prone to convergence issues, by proposing a two-stage framework with progressive training strategies, resulting in winning two champions and a runner-up in the NTIRE 2022 challenges.
As a widely studied task, video restoration aims to enhance the quality of the videos with multiple potential degradations, such as noises, blurs and compression artifacts. Among video restorations, compressed video quality enhancement and video super-resolution are two of the main tacks with significant values in practical scenarios. Recently, recurrent neural networks and transformers attract increasing research interests in this field, due to their impressive capability in sequence-to-sequence modeling. However, the training of these models is not only costly but also relatively hard to converge, with gradient exploding and vanishing problems. To cope with these problems, we proposed a two-stage framework including a multi-frame recurrent network and a single-frame transformer. Besides, multiple training strategies, such as transfer learning and progressive training, are developed to shorten the training time and improve the model performance. Benefiting from the above technical contributions, our solution wins two champions and a runner-up in the NTIRE 2022 super-resolution and quality enhancement of compressed video challenges. Code is available at https://github.com/ryanxingql/winner-ntire22-vqe.