Localizing Object-level Shape Variations with Text-to-Image Diffusion Models
This work addresses a specific bottleneck in text-to-image workflows for users needing fine-grained control over object shapes, representing an incremental improvement in manipulation techniques.
The paper tackles the problem of enabling object-level shape exploration in text-to-image generation by introducing a method to generate variations in the shape of specific objects while preserving semantics, achieving effective localization and plausible results as demonstrated through extensive comparisons.
Text-to-image models give rise to workflows which often begin with an exploration step, where users sift through a large collection of generated images. The global nature of the text-to-image generation process prevents users from narrowing their exploration to a particular object in the image. In this paper, we present a technique to generate a collection of images that depicts variations in the shape of a specific object, enabling an object-level shape exploration process. Creating plausible variations is challenging as it requires control over the shape of the generated object while respecting its semantics. A particular challenge when generating object variations is accurately localizing the manipulation applied over the object's shape. We introduce a prompt-mixing technique that switches between prompts along the denoising process to attain a variety of shape choices. To localize the image-space operation, we present two techniques that use the self-attention layers in conjunction with the cross-attention layers. Moreover, we show that these localization techniques are general and effective beyond the scope of generating object variations. Extensive results and comparisons demonstrate the effectiveness of our method in generating object variations, and the competence of our localization techniques.