Generating Novel Scene Compositions from Single Images and Videos
This addresses the challenge of overfitting and memorization in GANs for computer vision applications, enabling creative scene generation from minimal data, though it is incremental as it builds on existing single image GANs.
The authors tackled the problem of training generative adversarial networks (GANs) in low-data regimes, such as from a single image or video clip, by introducing SIV-GAN, which generates novel scene compositions with more diverse and higher quality images compared to previous methods.
Given a large dataset for training, generative adversarial networks (GANs) can achieve remarkable performance for the image synthesis task. However, training GANs in extremely low data regimes remains a challenge, as overfitting often occurs, leading to memorization or training divergence. In this work, we introduce SIV-GAN, an unconditional generative model that can generate new scene compositions from a single training image or a single video clip. We propose a two-branch discriminator architecture, with content and layout branches designed to judge internal content and scene layout realism separately from each other. This discriminator design enables synthesis of visually plausible, novel compositions of a scene, with varying content and layout, while preserving the context of the original sample. Compared to previous single image GANs, our model generates more diverse, higher quality images, while not being restricted to a single image setting. We further introduce a new challenging task of learning from a few frames of a single video. In this training setup the training images are highly similar to each other, which makes it difficult for prior GAN models to achieve a synthesis of both high quality and diversity.