StoryImager: A Unified and Efficient Framework for Coherent Story Visualization and Completion
This work addresses the challenge of story visualization for applications in creative media and AI-assisted storytelling, offering a more efficient and unified solution, though it is incremental in building upon pre-trained text-to-image models.
The paper tackles the problem of generating coherent image sequences from storylines by addressing limitations in existing models, such as unidirectional generation and high computational cost, and proposes StoryImager, a bidirectional and efficient framework that achieves state-of-the-art performance with improved metrics.
Story visualization aims to generate a series of realistic and coherent images based on a storyline. Current models adopt a frame-by-frame architecture by transforming the pre-trained text-to-image model into an auto-regressive manner. Although these models have shown notable progress, there are still three flaws. 1) The unidirectional generation of auto-regressive manner restricts the usability in many scenarios. 2) The additional introduced story history encoders bring an extremely high computational cost. 3) The story visualization and continuation models are trained and inferred independently, which is not user-friendly. To these ends, we propose a bidirectional, unified, and efficient framework, namely StoryImager. The StoryImager enhances the storyboard generative ability inherited from the pre-trained text-to-image model for a bidirectional generation. Specifically, we introduce a Target Frame Masking Strategy to extend and unify different story image generation tasks. Furthermore, we propose a Frame-Story Cross Attention Module that decomposes the cross attention for local fidelity and global coherence. Moreover, we design a Contextual Feature Extractor to extract contextual information from the whole storyline. The extensive experimental results demonstrate the excellent performance of our StoryImager. The code is available at https://github.com/tobran/StoryImager.