Stochastic Image-to-Video Synthesis using cINNs
This addresses video understanding and synthesis for applications like content creation, but it is incremental as it builds on existing cINN methods for a specific task.
The paper tackles the problem of stochastic image-to-video synthesis by proposing a bijective mapping between videos and static content plus residual information, using a conditional invertible neural network (cINN) to enable controlled video generation, with experiments on four datasets showing effectiveness in quality and diversity.
Video understanding calls for a model to learn the characteristic interplay between static scene content and its dynamics: Given an image, the model must be able to predict a future progression of the portrayed scene and, conversely, a video should be explained in terms of its static image content and all the remaining characteristics not present in the initial frame. This naturally suggests a bijective mapping between the video domain and the static content as well as residual information. In contrast to common stochastic image-to-video synthesis, such a model does not merely generate arbitrary videos progressing the initial image. Given this image, it rather provides a one-to-one mapping between the residual vectors and the video with stochastic outcomes when sampling. The approach is naturally implemented using a conditional invertible neural network (cINN) that can explain videos by independently modelling static and other video characteristics, thus laying the basis for controlled video synthesis. Experiments on four diverse video datasets demonstrate the effectiveness of our approach in terms of both the quality and diversity of the synthesized results. Our project page is available at https://bit.ly/3t66bnU.