Learning to Model Multimodal Semantic Alignment for Story Visualization
This addresses the challenge of maintaining global consistency in generated image sequences for story visualization, which is important for applications in creative content generation and multimodal AI, though it appears to be an incremental improvement over existing GAN-based approaches.
The paper tackles the problem of semantic misalignment in story visualization, where current methods struggle with generating consistent images from multi-sentence stories, and proposes a method that learns to dynamically match semantic levels between text and image representations, achieving improvements in image quality and story consistency compared to state-of-the-art methods.
Story visualization aims to generate a sequence of images to narrate each sentence in a multi-sentence story, where the images should be realistic and keep global consistency across dynamic scenes and characters. Current works face the problem of semantic misalignment because of their fixed architecture and diversity of input modalities. To address this problem, we explore the semantic alignment between text and image representations by learning to match their semantic levels in the GAN-based generative model. More specifically, we introduce dynamic interactions according to learning to dynamically explore various semantic depths and fuse the different-modal information at a matched semantic level, which thus relieves the text-image semantic misalignment problem. Extensive experiments on different datasets demonstrate the improvements of our approach, neither using segmentation masks nor auxiliary captioning networks, on image quality and story consistency, compared with state-of-the-art methods.