SSG-Dit: A Spatial Signal Guided Framework for Controllable Video Generation
This work addresses a key challenge in video generation for users needing precise control over content, though it appears incremental as it builds on existing diffusion transformer methods.
The paper tackles the problem of maintaining semantic consistency in controllable video generation by proposing SSG-DiT, a framework that uses spatial signal prompting and a lightweight adapter to guide a diffusion transformer, achieving state-of-the-art performance on metrics like spatial relationship control in the VBench benchmark.
Controllable video generation aims to synthesize video content that aligns precisely with user-provided conditions, such as text descriptions and initial images. However, a significant challenge persists in this domain: existing models often struggle to maintain strong semantic consistency, frequently generating videos that deviate from the nuanced details specified in the prompts. To address this issue, we propose SSG-DiT (Spatial Signal Guided Diffusion Transformer), a novel and efficient framework for high-fidelity controllable video generation. Our approach introduces a decoupled two-stage process. The first stage, Spatial Signal Prompting, generates a spatially aware visual prompt by leveraging the rich internal representations of a pre-trained multi-modal model. This prompt, combined with the original text, forms a joint condition that is then injected into a frozen video DiT backbone via our lightweight and parameter-efficient SSG-Adapter. This unique design, featuring a dual-branch attention mechanism, allows the model to simultaneously harness its powerful generative priors while being precisely steered by external spatial signals. Extensive experiments demonstrate that SSG-DiT achieves state-of-the-art performance, outperforming existing models on multiple key metrics in the VBench benchmark, particularly in spatial relationship control and overall consistency.