DTVNet+: A High-Resolution Scenic Dataset for Dynamic Time-lapse Video Generation
This addresses the challenge of creating diverse time-lapse videos for scenic content generation, though it appears incremental as it builds on existing video generation techniques with specific adaptations.
The paper tackles the problem of generating dynamic time-lapse videos from a single landscape image by proposing DTVNet, an end-to-end framework that uses normalized motion vectors, and introduces a high-resolution dataset called Quick-Sky-Time for evaluation. The results show superiority over state-of-the-art methods in generating high-quality and varied videos.
This paper presents a novel end-to-end dynamic time-lapse video generation framework, named DTVNet, to generate diversified time-lapse videos from a single landscape image conditioned on normalized motion vectors. The proposed DTVNet consists of two submodules: \emph{Optical Flow Encoder} (OFE) and \emph{Dynamic Video Generator} (DVG). The OFE maps a sequence of optical flow maps to a \emph{normalized motion vector} that encodes the motion information of the generated video. The DVG contains motion and content streams to learn from the motion vector and the single landscape image. Besides, it contains an encoder to learn shared content features and a decoder to construct video frames with corresponding motion. Specifically, the \emph{motion stream} introduces multiple \emph{adaptive instance normalization} (AdaIN) layers to integrate multi-level motion information for controlling the object motion. In the testing stage, videos with the same content but various motion information can be generated by different \emph{normalized motion vectors} based on only one input image. Also, we propose a high-resolution scenic time-lapse video dataset, named Quick-Sky-Time, to evaluate different approaches, which can be viewed as a new benchmark for high-quality scenic image and video generation tasks. We further conduct experiments on Sky Time-lapse, Beach, and Quick-Sky-Time datasets. The results demonstrate the superiority of our approach over state-of-the-art methods for generating high-quality and various dynamic videos.