Multimodal Conditional Image Synthesis with Product-of-Experts GANs
This addresses a practical limitation in image synthesis for users who need to leverage multimodal inputs, representing a strong specific gain rather than a foundational advance.
The paper tackles the problem of generating images from multiple input modalities (e.g., text, sketches) by proposing PoE-GAN, which synthesizes high-quality and diverse images conditioned on any combination of modalities, outperforming both multimodal and unimodal state-of-the-art methods.
Existing conditional image synthesis frameworks generate images based on user inputs in a single modality, such as text, segmentation, sketch, or style reference. They are often unable to leverage multimodal user inputs when available, which reduces their practicality. To address this limitation, we propose the Product-of-Experts Generative Adversarial Networks (PoE-GAN) framework, which can synthesize images conditioned on multiple input modalities or any subset of them, even the empty set. PoE-GAN consists of a product-of-experts generator and a multimodal multiscale projection discriminator. Through our carefully designed training scheme, PoE-GAN learns to synthesize images with high quality and diversity. Besides advancing the state of the art in multimodal conditional image synthesis, PoE-GAN also outperforms the best existing unimodal conditional image synthesis approaches when tested in the unimodal setting. The project website is available at https://deepimagination.github.io/PoE-GAN .