SIRfyN: Single Image Relighting from your Neighbors
This addresses the challenge of realistic image relighting for computer vision and graphics applications, though it appears incremental as it builds on established illumination theory with a novel estimation approach.
The paper tackles the problem of relighting a scene from a single image to change overall shading while producing natural-looking results, with applications in generating training data and authoring environments. It introduces a method that uses similar scenes to estimate lighting fields without requiring expensive datasets, achieving qualitative improvements like erasing soft shadows and quantitative evaluation via an extended FID metric.
We show how to relight a scene, depicted in a single image, such that (a) the overall shading has changed and (b) the resulting image looks like a natural image of that scene. Applications for such a procedure include generating training data and building authoring environments. Naive methods for doing this fail. One reason is that shading and albedo are quite strongly related; for example, sharp boundaries in shading tend to appear at depth discontinuities, which usually apparent in albedo. The same scene can be lit in different ways, and established theory shows the different lightings form a cone (the illumination cone). Novel theory shows that one can use similar scenes to estimate the different lightings that apply to a given scene, with bounded expected error. Our method exploits this theory to estimate a representation of the available lighting fields in the form of imputed generators of the illumination cone. Our procedure does not require expensive "inverse graphics" datasets, and sees no ground truth data of any kind. Qualitative evaluation suggests the method can erase and restore soft indoor shadows, and can "steer" light around a scene. We offer a summary quantitative evaluation of the method with a novel application of the FID. An extension of the FID allows per-generated-image evaluation. Furthermore, we offer qualitative evaluation with a user study, and show that our method produces images that can successfully be used for data augmentation.