UniMoCa: Unifying Motion and Camera Controls as Visual Proxies for Faithful Human Video Generation
This work is significant for researchers and developers in human video generation, as it offers a more robust and consistent method for controlling both human motion and camera trajectories, particularly in challenging multi-person and dynamic camera scenarios. It represents an incremental improvement over existing heterogeneous control interfaces.
The paper addresses the challenge of controlling human motion and camera movement in human-oriented video generation, especially in complex multi-person scenes. It proposes UniMoCa, a framework that unifies motion and camera controls into a single visual representation called Motion-Camera Visual Proxy (MCVP), leading to substantial gains in human motion control, camera control, temporal consistency, and camera-aware robustness.
Controlling human motion and camera movement is essential for faithful human-oriented video generation, yet remains challenging in multi-person scenes with large body motions, occlusions, and dynamic cameras. Existing pipelines typically rely on visual motion sequences, such as skeleton maps, pose maps, or rendered body representations, for motion control, while using camera embeddings for camera control. Such heterogeneous control interfaces force video generation models to reconcile pixel-aligned visual cues with non-visual geometric embeddings, making motion-camera attribution difficult and sensitive to camera estimation errors. We propose \textbf{UniMoCa}, a representation-driven framework that unifies motion and camera controls in visual space. At the core of UniMoCa is \textbf{Motion-Camera Visual Proxy} (\textbf{MCVP}), a mutually-sharable novel representation that converts 3D human motion and camera trajectories extracted from driving videos into an identity-neutral visual proxy. MCVP renders temporally aligned human geometry under the recovered camera trajectory and augments it with explicit camera trajectory markers, replacing heterogeneous visual-parametric controls with distinguishable visual cues. As both control factors are represented in the same visual space, they become mutually compatible rather than heterogeneous, enabling consistent joint reasoning and editing during video generation. We further curate a \textbf{MCVP-Video} dataset covering complex actions, multi-person interactions, and diverse camera trajectories. Experiments based on the Wan2.2 I2V show that UniMoCa achieves substantial gains in human motion control, camera control, temporal consistency, and camera-aware robustness with minimal additional complexity. More details are shown in our Project page: https://tanliming-daniel.github.io/UniMoCa/.