RePerformer: Immersive Human-centric Volumetric Videos from Playback to Photoreal Reperformance
This addresses the limitation of existing methods that focus only on replaying scenes or animating avatars, enabling more flexible immersive experiences for applications like virtual reality and entertainment.
The paper tackles the problem of creating immersive human-centric volumetric videos that can both replay existing scenes and be re-performed with novel motions, presenting RePerformer which achieves high-fidelity rendering and sets a new benchmark for this unified playback-then-reperformance paradigm.
Human-centric volumetric videos offer immersive free-viewpoint experiences, yet existing methods focus either on replaying general dynamic scenes or animating human avatars, limiting their ability to re-perform general dynamic scenes. In this paper, we present RePerformer, a novel Gaussian-based representation that unifies playback and re-performance for high-fidelity human-centric volumetric videos. Specifically, we hierarchically disentangle the dynamic scenes into motion Gaussians and appearance Gaussians which are associated in the canonical space. We further employ a Morton-based parameterization to efficiently encode the appearance Gaussians into 2D position and attribute maps. For enhanced generalization, we adopt 2D CNNs to map position maps to attribute maps, which can be assembled into appearance Gaussians for high-fidelity rendering of the dynamic scenes. For re-performance, we develop a semantic-aware alignment module and apply deformation transfer on motion Gaussians, enabling photo-real rendering under novel motions. Extensive experiments validate the robustness and effectiveness of RePerformer, setting a new benchmark for playback-then-reperformance paradigm in human-centric volumetric videos.