Multi-object Video Generation from Single Frame Layouts
This work addresses the challenge of practical video synthesis for applications requiring multi-object understanding, though it appears incremental as an adaptation from image generation methods.
The paper tackles the problem of generating multi-object videos from simplified single-frame layouts, addressing the limitations of existing methods that require complex annotations or focus on single objects. The proposed framework demonstrates effectiveness on two video recognition benchmarks compared to a baseline.
In this paper, we study video synthesis with emphasis on simplifying the generation conditions. Most existing video synthesis models or datasets are designed to address complex motions of a single object, lacking the ability of comprehensively understanding the spatio-temporal relationships among multiple objects. Besides, current methods are usually conditioned on intricate annotations (e.g. video segmentations) to generate new videos, being fundamentally less practical. These motivate us to generate multi-object videos conditioning exclusively on object layouts from a single frame. To solve above challenges and inspired by recent research on image generation from layouts, we have proposed a novel video generative framework capable of synthesizing global scenes with local objects, via implicit neural representations and layout motion self-inference. Our framework is a non-trivial adaptation from image generation methods, and is new to this field. In addition, our model has been evaluated on two widely-used video recognition benchmarks, demonstrating effectiveness compared to the baseline model.