Enhanced Deep Animation Video Interpolation
This addresses the specific challenge of adapting frame interpolation methods from natural to animation videos, which is incremental as it builds on existing algorithms like DAIN and ANIN.
The paper tackles the problem of frame interpolation for cartoon videos, which have low frame rates and nonlinear motion, by introducing AutoFI to automatically render synthetic training data that ensures linear motion, and SktFI for post-processing with user sketches, resulting in high perceptual quality in interpolated frames.
Existing learning-based frame interpolation algorithms extract consecutive frames from high-speed natural videos to train the model. Compared to natural videos, cartoon videos are usually in a low frame rate. Besides, the motion between consecutive cartoon frames is typically nonlinear, which breaks the linear motion assumption of interpolation algorithms. Thus, it is unsuitable for generating a training set directly from cartoon videos. For better adapting frame interpolation algorithms from nature video to animation video, we present AutoFI, a simple and effective method to automatically render training data for deep animation video interpolation. AutoFI takes a layered architecture to render synthetic data, which ensures the assumption of linear motion. Experimental results show that AutoFI performs favorably in training both DAIN and ANIN. However, most frame interpolation algorithms will still fail in error-prone areas, such as fast motion or large occlusion. Besides AutoFI, we also propose a plug-and-play sketch-based post-processing module, named SktFI, to refine the final results using user-provided sketches manually. With AutoFI and SktFI, the interpolated animation frames show high perceptual quality.