UPGPT: Universal Diffusion Model for Person Image Generation, Editing and Pose Transfer
This addresses the issue of inconsistent person appearance in generative AI for applications like pose transfer and editing, though it appears incremental as it builds on existing diffusion models.
The authors tackled the problem of appearance inconsistency in text-to-image models for person generation by proposing a multimodal diffusion model that accepts text, pose, and visual prompts, achieving a unified method for person image generation, pose transfer, and mask-less editing.
Text-to-image models (T2I) such as StableDiffusion have been used to generate high quality images of people. However, due to the random nature of the generation process, the person has a different appearance e.g. pose, face, and clothing, despite using the same text prompt. The appearance inconsistency makes T2I unsuitable for pose transfer. We address this by proposing a multimodal diffusion model that accepts text, pose, and visual prompting. Our model is the first unified method to perform all person image tasks - generation, pose transfer, and mask-less edit. We also pioneer using small dimensional 3D body model parameters directly to demonstrate new capability - simultaneous pose and camera view interpolation while maintaining the person's appearance.