DIRECT: Video Mashup Creation via Hierarchical Multi-Agent Planning and Intent-Guided Editing
This addresses the challenge of creating professional-grade video mashups automatically, which is incremental as it builds on existing editing frameworks by focusing on multimodal orchestration.
The paper tackles the problem of automated video mashup creation, which often suffers from disjointed sequences due to poor cross-level multimodal orchestration, by proposing the DIRECT framework; it significantly outperforms state-of-the-art baselines in objective metrics and human evaluation.
Video mashup creation represents a complex video editing paradigm that recomposes existing footage to craft engaging audio-visual experiences, demanding intricate orchestration across semantic, visual, and auditory dimensions and multiple levels. However, existing automated editing frameworks often overlook the cross-level multimodal orchestration to achieve professional-grade fluidity, resulting in disjointed sequences with abrupt visual transitions and musical misalignment. To address this, we formulate video mashup creation as a Multimodal Coherency Satisfaction Problem (MMCSP) and propose the DIRECT framework. Simulating a professional production pipeline, our hierarchical multi-agent framework decomposes the challenge into three cascade levels: the Screenwriter for source-aware global structural anchoring, the Director for instantiating adaptive editing intent and guidance, and the Editor for intent-guided shot sequence editing with fine-grained optimization. We further introduce Mashup-Bench, a comprehensive benchmark with tailored metrics for visual continuity and auditory alignment. Extensive experiments demonstrate that DIRECT significantly outperforms state-of-the-art baselines in both objective metrics and human subjective evaluation. Project page and code: https://github.com/AK-DREAM/DIRECT