Aether Weaver: Multimodal Affective Narrative Co-Generation with Dynamic Scene Graphs
This work addresses the challenge of creating immersive storytelling experiences for creative prototyping, though it appears incremental as it builds on existing multimodal generation techniques.
The paper tackles the problem of multimodal narrative generation by introducing Aether Weaver, an integrated framework that concurrently synthesizes text, scene graphs, visuals, and soundscapes, which significantly enhances narrative depth, visual fidelity, and emotional resonance compared to baseline approaches.
We introduce Aether Weaver, a novel, integrated framework for multimodal narrative co-generation that overcomes limitations of sequential text-to-visual pipelines. Our system concurrently synthesizes textual narratives, dynamic scene graph representations, visual scenes, and affective soundscapes, driven by a tightly integrated, co-generation mechanism. At its core, the Narrator, a large language model, generates narrative text and multimodal prompts, while the Director acts as a dynamic scene graph manager, and analyzes the text to build and maintain a structured representation of the story's world, ensuring spatio-temporal and relational consistency for visual rendering and subsequent narrative generation. Additionally, a Narrative Arc Controller guides the high-level story structure, influencing multimodal affective consistency, further complemented by an Affective Tone Mapper that ensures congruent emotional expression across all modalities. Through qualitative evaluations on a diverse set of narrative prompts encompassing various genres, we demonstrate that Aether Weaver significantly enhances narrative depth, visual fidelity, and emotional resonance compared to cascaded baseline approaches. This integrated framework provides a robust platform for rapid creative prototyping and immersive storytelling experiences.