Time-series Initialization and Conditioning for Video-agnostic Stabilization of Video Super-Resolution using Recurrent Networks
This addresses a domain gap issue in VSR for long videos, offering a more stable training approach, though it is incremental as it builds on existing RNN-based VSR methods.
The paper tackles the degradation in Video Super-Resolution (VSR) for long videos when using Recurrent Neural Networks (RNNs) trained on short clips, proposing a training strategy that stabilizes VSR by varying RNN hidden states based on video properties and using frame-number conditioning, with experimental results showing it outperforms base methods across videos of different lengths and dynamics.
A Recurrent Neural Network (RNN) for Video Super Resolution (VSR) is generally trained with randomly clipped and cropped short videos extracted from original training videos due to various challenges in learning RNNs. However, since this RNN is optimized to super-resolve short videos, VSR of long videos is degraded due to the domain gap. Our preliminary experiments reveal that such degradation changes depending on the video properties, such as the video length and dynamics. To avoid this degradation, this paper proposes the training strategy of RNN for VSR that can work efficiently and stably independently of the video length and dynamics. The proposed training strategy stabilizes VSR by training a VSR network with various RNN hidden states changed depending on the video properties. Since computing such a variety of hidden states is time-consuming, this computational cost is reduced by reusing the hidden states for efficient training. In addition, training stability is further improved with frame-number conditioning. Our experimental results demonstrate that the proposed method performed better than base methods in videos with various lengths and dynamics.