Image2Gif: Generating Continuous Realistic Animations with Warping NODEs
This addresses the need for high-quality video frame interpolation in vision applications, such as increasing frame rates or generating facial emotion motions, but it is incremental as it builds on existing neural ODE and interpolation methods.
The paper tackles the problem of generating smooth, realistic animations from only two input frames by proposing Warping Neural ODE, a framework that uses continuous spatial transformations based on differential equations to achieve smoothness with infinitely small time steps, showing applications in video frame interpolation with GAN and L2 loss training.
Generating smooth animations from a limited number of sequential observations has a number of applications in vision. For example, it can be used to increase number of frames per second, or generating a new trajectory only based on first and last frames, e.g. a motion of face emotions. Despite the discrete observed data (frames), the problem of generating a new trajectory is a continues problem. In addition, to be perceptually realistic, the domain of an image should not alter drastically through the trajectory of changes. In this paper, we propose a new framework, Warping Neural ODE, for generating a smooth animation (video frame interpolation) in a continuous manner, given two ("farther apart") frames, denoting the start and the end of the animation. The key feature of our framework is utilizing the continuous spatial transformation of the image based on the vector field, derived from a system of differential equations. This allows us to achieve the smoothness and the realism of an animation with infinitely small time steps between the frames. We show the application of our work in generating an animation given two frames, in different training settings, including Generative Adversarial Network (GAN) and with $L_2$ loss.