Directed Diffusion: Direct Control of Object Placement through Attention Guidance
This addresses a crucial capability gap in storytelling applications, such as film and animation, by enabling directed placement of characters and objects across images, though it is an incremental improvement building on existing pre-trained models.
The paper tackled the problem of text-guided diffusion models struggling to compose scenes with multiple objects in specified positional relationships by introducing an optimization objective that guides cross-attention maps to place objects at desired positions, resulting in a method that provides easy high-level control over object placement while maintaining coherence with the background.
Text-guided diffusion models such as DALLE-2, Imagen, eDiff-I, and Stable Diffusion are able to generate an effectively endless variety of images given only a short text prompt describing the desired image content. In many cases the images are of very high quality. However, these models often struggle to compose scenes containing several key objects such as characters in specified positional relationships. The missing capability to ``direct'' the placement of characters and objects both within and across images is crucial in storytelling, as recognized in the literature on film and animation theory. In this work, we take a particularly straightforward approach to providing the needed direction. Drawing on the observation that the cross-attention maps for prompt words reflect the spatial layout of objects denoted by those words, we introduce an optimization objective that produces ``activation'' at desired positions in these cross-attention maps. The resulting approach is a step toward generalizing the applicability of text-guided diffusion models beyond single images to collections of related images, as in storybooks. Directed Diffusion provides easy high-level positional control over multiple objects, while making use of an existing pre-trained model and maintaining a coherent blend between the positioned objects and the background. Moreover, it requires only a few lines to implement.