Stochastic Dynamics for Video Infilling
This addresses the problem of video infilling for applications requiring long-interval frame generation, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles video infilling for generating plausible frame sequences over long intervals, introducing the Stochastic Dynamics Video Infilling (SDVI) framework that models this as a constrained stochastic generation process. Experimental results show SDVI generates clear sequences with realistic motions that smoothly transition between start and terminal frames.
In this paper, we introduce a stochastic dynamics video infilling (SDVI) framework to generate frames between long intervals in a video. Our task differs from video interpolation which aims to produce transitional frames for a short interval between every two frames and increase the temporal resolution. Our task, namely video infilling, however, aims to infill long intervals with plausible frame sequences. Our framework models the infilling as a constrained stochastic generation process and sequentially samples dynamics from the inferred distribution. SDVI consists of two parts: (1) a bi-directional constraint propagation module to guarantee the spatial-temporal coherence among frames, (2) a stochastic sampling process to generate dynamics from the inferred distributions. Experimental results show that SDVI can generate clear frame sequences with varying contents. Moreover, motions in the generated sequence are realistic and able to transfer smoothly from the given start frame to the terminal frame. Our project site is https://xharlie.github.io/projects/project_sites/SDVI/video_results.html