Facilitating Video Story Interaction with Multi-Agent Collaborative System
This addresses the need for more personalized and interactive video narratives for viewers, though it appears incremental as it builds on existing VLM and MAS techniques.
The paper tackles the problem of limited customization and user engagement in video story interaction by proposing a multi-agent collaborative system that uses a Vision Language Model, Retrieval-Augmented Generation, and Multi-Agent System to create evolving characters and scenes, applied to the Harry Potter series to effectively portray emergent character behavior and growth.
Video story interaction enables viewers to engage with and explore narrative content for personalized experiences. However, existing methods are limited to user selection, specially designed narratives, and lack customization. To address this, we propose an interactive system based on user intent. Our system uses a Vision Language Model (VLM) to enable machines to understand video stories, combining Retrieval-Augmented Generation (RAG) and a Multi-Agent System (MAS) to create evolving characters and scene experiences. It includes three stages: 1) Video story processing, utilizing VLM and prior knowledge to simulate human understanding of stories across three modalities. 2) Multi-space chat, creating growth-oriented characters through MAS interactions based on user queries and story stages. 3) Scene customization, expanding and visualizing various story scenes mentioned in dialogue. Applied to the Harry Potter series, our study shows the system effectively portrays emergent character social behavior and growth, enhancing the interactive experience in the video story world.