DOS: Directional Object Separation in Text Embeddings for Multi-Object Image Generation
This addresses a key limitation in text-to-image generation for users needing accurate multi-object scenes, though it is an incremental improvement on existing methods.
The paper tackles the problem of multi-object image generation in text-to-image models, where object neglect or mixing occurs, by proposing DOS to modify CLIP text embeddings, resulting in a 26.24%-43.04% improvement in human evaluation votes over competing methods.
Recent progress in text-to-image (T2I) generative models has led to significant improvements in generating high-quality images aligned with text prompts. However, these models still struggle with prompts involving multiple objects, often resulting in object neglect or object mixing. Through extensive studies, we identify four problematic scenarios, Similar Shapes, Similar Textures, Dissimilar Background Biases, and Many Objects, where inter-object relationships frequently lead to such failures. Motivated by two key observations about CLIP embeddings, we propose DOS (Directional Object Separation), a method that modifies three types of CLIP text embeddings before passing them into text-to-image models. Experimental results show that DOS consistently improves the success rate of multi-object image generation and reduces object mixing. In human evaluations, DOS significantly outperforms four competing methods, receiving 26.24%-43.04% more votes across four benchmarks. These results highlight DOS as a practical and effective solution for improving multi-object image generation.