Video Frame Interpolation Transformer
This work improves video interpolation for applications like video editing and compression, though it is incremental as it adapts Transformers to an existing task.
The authors tackled the problem of video frame interpolation by addressing limitations of convolutional neural networks, such as content-agnostic kernels and restricted receptive fields, and proposed a Transformer-based framework that achieved state-of-the-art performance on multiple benchmark datasets.
Existing methods for video interpolation heavily rely on deep convolution neural networks, and thus suffer from their intrinsic limitations, such as content-agnostic kernel weights and restricted receptive field. To address these issues, we propose a Transformer-based video interpolation framework that allows content-aware aggregation weights and considers long-range dependencies with the self-attention operations. To avoid the high computational cost of global self-attention, we introduce the concept of local attention into video interpolation and extend it to the spatial-temporal domain. Furthermore, we propose a space-time separation strategy to save memory usage, which also improves performance. In addition, we develop a multi-scale frame synthesis scheme to fully realize the potential of Transformers. Extensive experiments demonstrate the proposed model performs favorably against the state-of-the-art methods both quantitatively and qualitatively on a variety of benchmark datasets.