Generating Fine Details of Entity Interactions
This addresses a specific bottleneck in text-to-image generation for producing realistic multi-entity interactions, with incremental improvements to existing models.
The paper tackles the challenge of generating high-fidelity images depicting accurate interactions between multiple entities, which existing text-to-image models struggle with due to scarce training data. They introduce the InterActing dataset with 1000 fine-grained prompts and propose DetailScribe, a method that decomposes interactions and refines diffusion-generated images, showing significantly improved image quality in evaluations.
Images not only depict objects but also encapsulate rich interactions between them. However, generating faithful and high-fidelity images involving multiple entities interacting with each other, is a long-standing challenge. While pre-trained text-to-image models are trained on large-scale datasets to follow diverse text instructions, they struggle to generate accurate interactions, likely due to the scarcity of training data for uncommon object interactions. This paper introduces InterActing, an interaction-focused dataset with 1000 fine-grained prompts covering three key scenarios: (1) functional and action-based interactions, (2) compositional spatial relationships, and (3) multi-subject interactions. To address interaction generation challenges, we propose a decomposition-augmented refinement procedure. Our approach, DetailScribe, built on Stable Diffusion 3.5, leverages LLMs to decompose interactions into finer-grained concepts, uses a VLM to critique generated images, and applies targeted interventions within the diffusion process in refinement. Automatic and human evaluations show significantly improved image quality, demonstrating the potential of enhanced inference strategies. Our dataset and code are available at https://concepts-ai.com/p/detailscribe/ to facilitate future exploration of interaction-rich image generation.