Adaptive Visual Conditioning for Semantic Consistency in Diffusion-Based Story Continuation
This work addresses story continuation for AI-generated narratives, offering an incremental improvement in handling conflicting visual and textual inputs.
The paper tackled the problem of generating coherent next images in narrative sequences by introducing Adaptive Visual Conditioning (AVC), a diffusion-based framework that selectively uses prior visual context and improves data quality, achieving superior coherence and semantic consistency compared to baselines in challenging cases.
Story continuation focuses on generating the next image in a narrative sequence so that it remains coherent with both the ongoing text description and the previously observed images. A central challenge in this setting lies in utilizing prior visual context effectively, while ensuring semantic alignment with the current textual input. In this work, we introduce AVC (Adaptive Visual Conditioning), a framework for diffusion-based story continuation. AVC employs the CLIP model to retrieve the most semantically aligned image from previous frames. Crucially, when no sufficiently relevant image is found, AVC adaptively restricts the influence of prior visuals to only the early stages of the diffusion process. This enables the model to exploit visual context when beneficial, while avoiding the injection of misleading or irrelevant information. Furthermore, we improve data quality by re-captioning a noisy dataset using large language models, thereby strengthening textual supervision and semantic alignment. Quantitative results and human evaluations demonstrate that AVC achieves superior coherence, semantic consistency, and visual fidelity compared to strong baselines, particularly in challenging cases where prior visuals conflict with the current input.