Factorized and Controllable Neural Re-Rendering of Outdoor Scene for Photo Extrapolation
This addresses a specific challenge in photography applications for users wanting to expand tourist photos, though it is incremental as it builds on existing photo extrapolation and neural rendering techniques.
The paper tackles the problem of extrapolating a narrow field-of-view tourist photo to a wider scene while maintaining visual style, achieving photorealistic results through a factorized neural re-rendering model that enables applications like controllable re-rendering and 3D photo generation.
Expanding an existing tourist photo from a partially captured scene to a full scene is one of the desired experiences for photography applications. Although photo extrapolation has been well studied, it is much more challenging to extrapolate a photo (i.e., selfie) from a narrow field of view to a wider one while maintaining a similar visual style. In this paper, we propose a factorized neural re-rendering model to produce photorealistic novel views from cluttered outdoor Internet photo collections, which enables the applications including controllable scene re-rendering, photo extrapolation and even extrapolated 3D photo generation. Specifically, we first develop a novel factorized re-rendering pipeline to handle the ambiguity in the decomposition of geometry, appearance and illumination. We also propose a composited training strategy to tackle the unexpected occlusion in Internet images. Moreover, to enhance photo-realism when extrapolating tourist photographs, we propose a novel realism augmentation process to complement appearance details, which automatically propagates the texture details from a narrow captured photo to the extrapolated neural rendered image. The experiments and photo editing examples on outdoor scenes demonstrate the superior performance of our proposed method in both photo-realism and downstream applications.