DreamingComics: A Story Visualization Pipeline via Subject and Layout Customized Generation using Video Models
This addresses the challenge of consistent subject positioning and style in story visualization for applications like comics or visual storytelling, representing a strong incremental improvement with specific gains.
The paper tackles the problem of maintaining artistic consistency in story visualization by introducing DreamingComics, a layout-aware framework that uses a pretrained video diffusion-transformer model and novel techniques like RegionalRoPE and masked condition loss, resulting in a 29.2% increase in character consistency and a 36.2% increase in style similarity compared to previous methods.
Current story visualization methods tend to position subjects solely by text and face challenges in maintaining artistic consistency. To address these limitations, we introduce DreamingComics, a layout-aware story visualization framework. We build upon a pretrained video diffusion-transformer (DiT) model, leveraging its spatiotemporal priors to enhance identity and style consistency. For layout-based position control, we propose RegionalRoPE, a region-aware positional encoding scheme that re-indexes embeddings based on the target layout. Additionally, we introduce a masked condition loss to further constrain each subject's visual features to their designated region. To infer layouts from natural language scripts, we integrate an LLM-based layout generator trained to produce comic-style layouts, enabling flexible and controllable layout conditioning. We present a comprehensive evaluation of our approach, showing a 29.2% increase in character consistency and a 36.2% increase in style similarity compared to previous methods, while displaying high spatial accuracy. Our project page is available at https://yj7082126.github.io/dreamingcomics/