DynASyn: Multi-Subject Personalization Enabling Dynamic Action Synthesis
This addresses the challenge of dynamic action synthesis in multi-subject personalization for image generation, though it appears incremental as it builds on existing personalization methods.
The paper tackled the problem of personalizing multiple subjects in text-to-image diffusion models from a single reference image, which often leads to overfitting and struggles with modifying behaviors or dynamic interactions, and resulted in DynASyn, a method that outperforms baselines by synthesizing realistic images with novel contexts and dynamic interactions.
Recent advances in text-to-image diffusion models spurred research on personalization, i.e., a customized image synthesis, of subjects within reference images. Although existing personalization methods are able to alter the subjects' positions or to personalize multiple subjects simultaneously, they often struggle to modify the behaviors of subjects or their dynamic interactions. The difficulty is attributable to overfitting to reference images, which worsens if only a single reference image is available. We propose DynASyn, an effective multi-subject personalization from a single reference image addressing these challenges. DynASyn preserves the subject identity in the personalization process by aligning concept-based priors with subject appearances and actions. This is achieved by regularizing the attention maps between the subject token and images through concept-based priors. In addition, we propose concept-based prompt-and-image augmentation for an enhanced trade-off between identity preservation and action diversity. We adopt an SDE-based editing guided by augmented prompts to generate diverse appearances and actions while maintaining identity consistency in the augmented images. Experiments show that DynASyn is capable of synthesizing highly realistic images of subjects with novel contexts and dynamic interactions with the surroundings, and outperforms baseline methods in both quantitative and qualitative aspects.