PersonificationNet: Making customized subject act like a person
This addresses a lack of pose control in customized generation for users needing personalized content, but it is incremental as it builds on existing customization techniques.
The paper tackles the problem of fine-grained control over customized image generation, enabling a specified subject like a cartoon character to mimic a person's pose from a reference image, and reports that PersonificationNet outperforms state-of-the-art methods.
Recently customized generation has significant potential, which uses as few as 3-5 user-provided images to train a model to synthesize new images of a specified subject. Though subsequent applications enhance the flexibility and diversity of customized generation, fine-grained control over the given subject acting like the person's pose is still lack of study. In this paper, we propose a PersonificationNet, which can control the specified subject such as a cartoon character or plush toy to act the same pose as a given referenced person's image. It contains a customized branch, a pose condition branch and a structure alignment module. Specifically, first, the customized branch mimics specified subject appearance. Second, the pose condition branch transfers the body structure information from the human to variant instances. Last, the structure alignment module bridges the structure gap between human and specified subject in the inference stage. Experimental results show our proposed PersonificationNet outperforms the state-of-the-art methods.