Boost Video Frame Interpolation via Motion Adaptation
This addresses a domain-specific problem for video processing applications by improving interpolation accuracy for diverse motions, though it is incremental as it builds on existing pre-trained models.
The paper tackles the limited generalization of video frame interpolation models to unseen motions by proposing an optimization-based method with a cycle-consistency adaptation strategy and lightweight adapter, achieving state-of-the-art performance on benchmarks.
Video frame interpolation (VFI) is a challenging task that aims to generate intermediate frames between two consecutive frames in a video. Existing learning-based VFI methods have achieved great success, but they still suffer from limited generalization ability due to the limited motion distribution of training datasets. In this paper, we propose a novel optimization-based VFI method that can adapt to unseen motions at test time. Our method is based on a cycle-consistency adaptation strategy that leverages the motion characteristics among video frames. We also introduce a lightweight adapter that can be inserted into the motion estimation module of existing pre-trained VFI models to improve the efficiency of adaptation. Extensive experiments on various benchmarks demonstrate that our method can boost the performance of two-frame VFI models, outperforming the existing state-of-the-art methods, even those that use extra input.