People as Scene Probes
This addresses the challenge of realistic object insertion in video editing for users in graphics and visual effects, though it appears incremental as it builds on existing motion analysis and compositing techniques.
The paper tackles the problem of automatically compositing new objects into a scene from a single video by analyzing motion to infer depth, occlusion, lighting, and shadows, resulting in realistic integration with proper scaling, lighting, and shadow effects.
By analyzing the motion of people and other objects in a scene, we demonstrate how to infer depth, occlusion, lighting, and shadow information from video taken from a single camera viewpoint. This information is then used to composite new objects into the same scene with a high degree of automation and realism. In particular, when a user places a new object (2D cut-out) in the image, it is automatically rescaled, relit, occluded properly, and casts realistic shadows in the correct direction relative to the sun, and which conform properly to scene geometry. We demonstrate results (best viewed in supplementary video) on a range of scenes and compare to alternative methods for depth estimation and shadow compositing.