Generating Humanless Environment Walkthroughs from Egocentric Walking Tour Videos
For researchers and practitioners in environment modeling and computer vision, this work provides a method to clean up egocentric videos for downstream 3D/4D reconstruction, though it is incremental as it adapts an existing model with a new training dataset.
The paper addresses the problem of human presence in egocentric walking tour videos hindering environment modeling. They fine-tune the Casper video diffusion model on a semi-synthetic dataset to inpaint humans and shadows, achieving far better qualitative and quantitative performance than the baseline.
Egocentric "walking tour" videos provide a rich source of image data to develop rich and diverse visual models of environments around the world. However, the significant presence of humans in frames of these videos due to crowds and eye-level camera perspectives mitigates their usefulness in environment modeling applications. We focus on addressing this challenge by developing a generative algorithm that can realistically remove (i.e., inpaint) humans and their associated shadow effects from walking tour videos. Key to our approach is the construction of a rich semi-synthetic dataset of video clip pairs to train this generative model. Each pair in the dataset consists of an environment-only background clip, and a composite clip of walking humans with simulated shadows overlaid on the background. We randomly sourced both foreground and background components from real egocentric walking tour videos around the world to maintain visual diversity. We then used this dataset to fine-tune the state-of-the-art Casper video diffusion model for object and effects inpainting, and demonstrate that the resulting model performs far better than Casper both qualitatively and quantitatively at removing humans from walking tour clips with significant human presence and complex backgrounds. Finally, we show that the resulting generated clips can be used to build successful 3D/4D models of urban locations.