Synthesizing Environment-Specific People in Photographs
This addresses the need for realistic human synthesis in photographs for applications like image editing, but it is incremental as it builds on existing generation methods.
The paper tackles the problem of generating photo-realistic people with clothing appropriate for a given scene, achieving state-of-the-art performance in contextual full-body generation.
We present ESP, a novel method for context-aware full-body generation, that enables photo-realistic synthesis and inpainting of people wearing clothing that is semantically appropriate for the scene depicted in an input photograph. ESP is conditioned on a 2D pose and contextual cues that are extracted from the photograph of the scene and integrated into the generation process, where the clothing is modeled explicitly with human parsing masks (HPM). Generated HPMs are used as tight guiding masks for inpainting, such that no changes are made to the original background. Our models are trained on a dataset containing a set of in-the-wild photographs of people covering a wide range of different environments. The method is analyzed quantitatively and qualitatively, and we show that ESP outperforms the state-of-the-art on the task of contextual full-body generation.