Shape-Guided Diffusion with Inside-Outside Attention
This work addresses the need for more accurate user control in image generation for applications like editing, but it is incremental as it builds on existing diffusion models.
The paper tackles the problem of precise object silhouette control in text-to-image diffusion models by introducing Shape-Guided Diffusion with Inside-Outside Attention, achieving state-of-the-art results in shape faithfulness without degrading text alignment or image realism on a new ShapePrompts benchmark.
We introduce precise object silhouette as a new form of user control in text-to-image diffusion models, which we dub Shape-Guided Diffusion. Our training-free method uses an Inside-Outside Attention mechanism during the inversion and generation process to apply a shape constraint to the cross- and self-attention maps. Our mechanism designates which spatial region is the object (inside) vs. background (outside) then associates edits to the correct region. We demonstrate the efficacy of our method on the shape-guided editing task, where the model must replace an object according to a text prompt and object mask. We curate a new ShapePrompts benchmark derived from MS-COCO and achieve SOTA results in shape faithfulness without a degradation in text alignment or image realism according to both automatic metrics and annotator ratings. Our data and code will be made available at https://shape-guided-diffusion.github.io.