GenFusion: Feed-forward Human Performance Capture via Progressive Canonical Space Updates
This work addresses the challenge of novel view synthesis for dynamic humans from monocular video, a key problem in virtual reality and telepresence, by enabling temporally coherent appearance accumulation.
GenFusion introduces a feed-forward method for novel view synthesis of a human performer from monocular video, using a progressively updated canonical space to handle unseen regions. It achieves sharper reconstructions and plausible synthesis in unobserved areas, outperforming prior methods on 4D-Dress and MVHumanNet datasets.
We present a feed-forward human performance capture method that renders novel views of a performer from a monocular RGB stream. A key challenge in this setting is the lack of sufficient observations, especially for unseen regions. Assuming the subject moves continuously over time, we take advantage of the fact that more body parts become observable by maintaining a canonical space that is progressively updated with each incoming frame. This canonical space accumulates appearance information over time and serves as a context bank when direct observations are missing in the current live frame. To effectively utilize this context while respecting the deformation of the live state, we formulate the rendering process as probabilistic regression. This resolves conflicts between past and current observations, producing sharper reconstructions than deterministic regression approaches. Furthermore, it enables plausible synthesis even in regions with no prior observations. Experiments on in-domain (4D-Dress) and out-of-distribution (MVHumanNet) datasets demonstrate the effectiveness of our approach.