Multi-Tailed, Multi-Headed, Spatial Dynamic Memory refined Text-to-Image Synthesis
This work addresses the challenge of generating realistic images from text for applications like content creation, but it appears incremental as it builds on existing multi-stage refinement paradigms.
The paper tackles the problem of synthesizing high-quality images from text descriptions by addressing three limitations of existing multi-stage methods: entangled object attributes, uniform text interpretation across image regions, and single-shot refinement. The proposed MSMT-GAN method, which includes word-level initial generation, spatial dynamic memory, and iterative multi-headed refinement, achieves favorable performance against previous state-of-the-art on CUB and COCO datasets.
Synthesizing high-quality, realistic images from text-descriptions is a challenging task, and current methods synthesize images from text in a multi-stage manner, typically by first generating a rough initial image and then refining image details at subsequent stages. However, existing methods that follow this paradigm suffer from three important limitations. Firstly, they synthesize initial images without attempting to separate image attributes at a word-level. As a result, object attributes of initial images (that provide a basis for subsequent refinement) are inherently entangled and ambiguous in nature. Secondly, by using common text-representations for all regions, current methods prevent us from interpreting text in fundamentally different ways at different parts of an image. Different image regions are therefore only allowed to assimilate the same type of information from text at each refinement stage. Finally, current methods generate refinement features only once at each refinement stage and attempt to address all image aspects in a single shot. This single-shot refinement limits the precision with which each refinement stage can learn to improve the prior image. Our proposed method introduces three novel components to address these shortcomings: (1) An initial generation stage that explicitly generates separate sets of image features for each word n-gram. (2) A spatial dynamic memory module for refinement of images. (3) An iterative multi-headed mechanism to make it easier to improve upon multiple image aspects. Experimental results demonstrate that our Multi-Headed Spatial Dynamic Memory image refinement with our Multi-Tailed Word-level Initial Generation (MSMT-GAN) performs favourably against the previous state of the art on the CUB and COCO datasets.