Automatic Camera Trajectory Control with Enhanced Immersion for Virtual Cinematography
This addresses the problem of mastering cinematography for non-experts in virtual environments, though it is incremental as it builds on existing automatic methods by enhancing synchronization.
The paper tackles the challenge of creating immersive user-generated cinematic content by proposing a deep camera control framework that synchronizes camera movement with the actor's frame aesthetics, spatial action, and emotional status in 3D virtual stages, resulting in high-quality immersive cinematic videos as demonstrated quantitatively and qualitatively.
User-generated cinematic creations are gaining popularity as our daily entertainment, yet it is a challenge to master cinematography for producing immersive contents. Many existing automatic methods focus on roughly controlling predefined shot types or movement patterns, which struggle to engage viewers with the circumstances of the actor. Real-world cinematographic rules show that directors can create immersion by comprehensively synchronizing the camera with the actor. Inspired by this strategy, we propose a deep camera control framework that enables actor-camera synchronization in three aspects, considering frame aesthetics, spatial action, and emotional status in the 3D virtual stage. Following rule-of-thirds, our framework first modifies the initial camera placement to position the actor aesthetically. This adjustment is facilitated by a self-supervised adjustor that analyzes frame composition via camera projection. We then design a GAN model that can adversarially synthesize fine-grained camera movement based on the physical action and psychological state of the actor, using an encoder-decoder generator to map kinematics and emotional variables into camera trajectories. Moreover, we incorporate a regularizer to align the generated stylistic variances with specific emotional categories and intensities. The experimental results show that our proposed method yields immersive cinematic videos of high quality, both quantitatively and qualitatively. Live examples can be found in the supplementary video.