AdaCoF: Adaptive Collaboration of Flows for Video Frame Interpolation
This addresses complex motion handling in video processing, offering a generalized solution for real-world applications, though it is incremental in improving existing warping techniques.
The paper tackles video frame interpolation by proposing AdaCoF, a warping module that estimates kernel weights and offset vectors for each pixel, and it outperforms state-of-the-art methods on benchmarks like Middlebury.
Video frame interpolation is one of the most challenging tasks in video processing research. Recently, many studies based on deep learning have been suggested. Most of these methods focus on finding locations with useful information to estimate each output pixel using their own frame warping operations. However, many of them have Degrees of Freedom (DoF) limitations and fail to deal with the complex motions found in real world videos. To solve this problem, we propose a new warping module named Adaptive Collaboration of Flows (AdaCoF). Our method estimates both kernel weights and offset vectors for each target pixel to synthesize the output frame. AdaCoF is one of the most generalized warping modules compared to other approaches, and covers most of them as special cases of it. Therefore, it can deal with a significantly wide domain of complex motions. To further improve our framework and synthesize more realistic outputs, we introduce dual-frame adversarial loss which is applicable only to video frame interpolation tasks. The experimental results show that our method outperforms the state-of-the-art methods for both fixed training set environments and the Middlebury benchmark.