GUNet: A Graph Convolutional Network United Diffusion Model for Stable and Diversity Pose Generation
This work addresses the problem of enriching pose skeleton sources for pose-controllable image generation, which is incremental as it adapts diffusion models to a specific domain.
The paper tackles the challenge of generating diverse, structurally correct, and aesthetically pleasing human pose skeletons from natural language inputs by proposing PoseDiffusion, a framework based on a diffusion model with GUNet. Experimental results show it outperforms existing state-of-the-art methods in stability and diversity for text-driven pose skeleton generation.
Pose skeleton images are an important reference in pose-controllable image generation. In order to enrich the source of skeleton images, recent works have investigated the generation of pose skeletons based on natural language. These methods are based on GANs. However, it remains challenging to perform diverse, structurally correct and aesthetically pleasing human pose skeleton generation with various textual inputs. To address this problem, we propose a framework with GUNet as the main model, PoseDiffusion. It is the first generative framework based on a diffusion model and also contains a series of variants fine-tuned based on a stable diffusion model. PoseDiffusion demonstrates several desired properties that outperform existing methods. 1) Correct Skeletons. GUNet, a denoising model of PoseDiffusion, is designed to incorporate graphical convolutional neural networks. It is able to learn the spatial relationships of the human skeleton by introducing skeletal information during the training process. 2) Diversity. We decouple the key points of the skeleton and characterise them separately, and use cross-attention to introduce textual conditions. Experimental results show that PoseDiffusion outperforms existing SoTA algorithms in terms of stability and diversity of text-driven pose skeleton generation. Qualitative analyses further demonstrate its superiority for controllable generation in Stable Diffusion.