TV-Dialogue: Crafting Theme-Aware Video Dialogues with Immersive Interaction
This addresses the challenge of creating theme-aware and visually consistent dialogues for videos, with potential applications in video re-creation and film dubbing, though it appears incremental as it builds on existing LLM advancements.
The paper tackles the underexplored problem of video-based dialogue generation by introducing a novel task and framework that generates dialogues aligned with video content and user-specified themes, achieving this in a zero-shot manner without training for videos of any length and theme.
Recent advancements in LLMs have accelerated the development of dialogue generation across text and images, yet video-based dialogue generation remains underexplored and presents unique challenges. In this paper, we introduce Theme-aware Video Dialogue Crafting (TVDC), a novel task aimed at generating new dialogues that align with video content and adhere to user-specified themes. We propose TV-Dialogue, a novel multi-modal agent framework that ensures both theme alignment (i.e., the dialogue revolves around the theme) and visual consistency (i.e., the dialogue matches the emotions and behaviors of characters in the video) by enabling real-time immersive interactions among video characters, thereby accurately understanding the video content and generating new dialogue that aligns with the given themes. To assess the generated dialogues, we present a multi-granularity evaluation benchmark with high accuracy, interpretability and reliability, demonstrating the effectiveness of TV-Dialogue on self-collected dataset over directly using existing LLMs. Extensive experiments reveal that TV-Dialogue can generate dialogues for videos of any length and any theme in a zero-shot manner without training. Our findings underscore the potential of TV-Dialogue for various applications, such as video re-creation, film dubbing and its use in downstream multimodal tasks.