Controlling Human Shape and Pose in Text-to-Image Diffusion Models via Domain Adaptation
This work addresses the challenge of fine-tuning diffusion models for human animation tasks, offering a more cost-effective approach using synthetic data, though it is incremental as it builds on existing ControlNet architectures.
The authors tackled the problem of controlling human shape and pose in text-to-image diffusion models by using a 3D human parametric model (SMPL) and domain adaptation to handle synthetic data limitations, resulting in greater shape and pose diversity while maintaining visual fidelity and stability compared to a 2D pose-based method.
We present a methodology for conditional control of human shape and pose in pretrained text-to-image diffusion models using a 3D human parametric model (SMPL). Fine-tuning these diffusion models to adhere to new conditions requires large datasets and high-quality annotations, which can be more cost-effectively acquired through synthetic data generation rather than real-world data. However, the domain gap and low scene diversity of synthetic data can compromise the pretrained model's visual fidelity. We propose a domain-adaptation technique that maintains image quality by isolating synthetically trained conditional information in the classifier-free guidance vector and composing it with another control network to adapt the generated images to the input domain. To achieve SMPL control, we fine-tune a ControlNet-based architecture on the synthetic SURREAL dataset of rendered humans and apply our domain adaptation at generation time. Experiments demonstrate that our model achieves greater shape and pose diversity than the 2d pose-based ControlNet, while maintaining the visual fidelity and improving stability, proving its usefulness for downstream tasks such as human animation.