Latent-Reframe: Enabling Camera Control for Video Diffusion Model without Training
This addresses the need for efficient camera control in video generation for creators and researchers, reducing data and computational costs, though it is incremental as it builds on existing video diffusion models.
The paper tackled the problem of enabling precise camera pose control in video diffusion models without requiring fine-tuning, and achieved comparable or superior camera control precision and video quality to training-based methods.
Precise camera pose control is crucial for video generation with diffusion models. Existing methods require fine-tuning with additional datasets containing paired videos and camera pose annotations, which are both data-intensive and computationally costly, and can disrupt the pre-trained model distribution. We introduce Latent-Reframe, which enables camera control in a pre-trained video diffusion model without fine-tuning. Unlike existing methods, Latent-Reframe operates during the sampling stage, maintaining efficiency while preserving the original model distribution. Our approach reframes the latent code of video frames to align with the input camera trajectory through time-aware point clouds. Latent code inpainting and harmonization then refine the model latent space, ensuring high-quality video generation. Experimental results demonstrate that Latent-Reframe achieves comparable or superior camera control precision and video quality to training-based methods, without the need for fine-tuning on additional datasets.