Ctrl-VI: Controllable Video Synthesis via Variational Inference
This addresses the need for more flexible video synthesis workflows, though it is incremental as it builds on existing video generation backbones.
The paper tackles the problem of generating videos with mixed user controls of varying granularity, and results in improved controllability, diversity, and 3D consistency compared to prior methods.
Many video workflows benefit from a mixture of user controls with varying granularity, from exact 4D object trajectories and camera paths to coarse text prompts, while existing video generative models are typically trained for fixed input formats. We develop Ctrl-VI, a video synthesis method that addresses this need and generates samples with high controllability for specified elements while maintaining diversity for under-specified ones. We cast the task as variational inference to approximate a composed distribution, leveraging multiple video generation backbones to account for all task constraints collectively. To address the optimization challenge, we break down the problem into step-wise KL divergence minimization over an annealed sequence of distributions, and further propose a context-conditioned factorization technique that reduces modes in the solution space to circumvent local optima. Experiments suggest that our method produces samples with improved controllability, diversity, and 3D consistency compared to prior works.