Improving Personalized Image Generation through Social Context Feedback
This work addresses specific issues in personalized image generation for AI and creative applications, but it is incremental as it builds on existing methods with feedback modules.
The paper tackled limitations in personalized image generation, such as incorrect human poses, identity preservation, and unnatural gaze patterns, by using feedback-based fine-tuning with specialized detectors, resulting in improved interactions, facial identities, and image quality on three benchmark datasets.
Personalized image generation, where reference images of one or more subjects are used to generate their image according to a scene description, has gathered significant interest in the community. However, such generated images suffer from three major limitations -- complex activities, such as $<$man, pushing, motorcycle$>$ are not generated properly with incorrect human poses, reference human identities are not preserved, and generated human gaze patterns are unnatural/inconsistent with the scene description. In this work, we propose to overcome these shortcomings through feedback-based fine-tuning of existing personalized generation methods, wherein, state-of-art detectors of pose, human-object-interaction, human facial recognition and human gaze-point estimation are used to refine the diffusion model. We also propose timestep-based inculcation of different feedback modules, depending upon whether the signal is low-level (such as human pose), or high-level (such as gaze point). The images generated in this manner show an improvement in the generated interactions, facial identities and image quality over three benchmark datasets.