Virtually Being: Customizing Camera-Controllable Video Diffusion Models with Multi-View Performance Captures
This work addresses the challenge of integrating video generation into virtual production for applications like filmmaking and gaming, though it is incremental as it builds on existing video diffusion models.
The paper tackles the problem of achieving multi-view character consistency and 3D camera control in video diffusion models by introducing a customization data pipeline using volumetric captures and 4D Gaussian Splatting, resulting in improved video quality, higher personalization accuracy, and enhanced camera control and lighting adaptability.
We introduce a framework that enables both multi-view character consistency and 3D camera control in video diffusion models through a novel customization data pipeline. We train the character consistency component with recorded volumetric capture performances re-rendered with diverse camera trajectories via 4D Gaussian Splatting (4DGS), lighting variability obtained with a video relighting model. We fine-tune state-of-the-art open-source video diffusion models on this data to provide strong multi-view identity preservation, precise camera control, and lighting adaptability. Our framework also supports core capabilities for virtual production, including multi-subject generation using two approaches: joint training and noise blending, the latter enabling efficient composition of independently customized models at inference time; it also achieves scene and real-life video customization as well as control over motion and spatial layout during customization. Extensive experiments show improved video quality, higher personalization accuracy, and enhanced camera control and lighting adaptability, advancing the integration of video generation into virtual production. Our project page is available at: https://eyeline-labs.github.io/Virtually-Being.