VersaGen: Unleashing Versatile Visual Control for Text-to-Image Synthesis
This work addresses the problem of limited creative control for users in text-to-image synthesis, offering a more versatile approach, though it appears incremental by building on existing methods.
The paper tackles the challenge of enabling precise visual control in text-to-image synthesis by introducing VersaGen, a generative AI agent that supports versatile controls like single/multiple subjects and scene backgrounds, achieving validated effectiveness and flexibility through experiments on COCO and Sketchy datasets.
Despite the rapid advancements in text-to-image (T2I) synthesis, enabling precise visual control remains a significant challenge. Existing works attempted to incorporate multi-facet controls (text and sketch), aiming to enhance the creative control over generated images. However, our pilot study reveals that the expressive power of humans far surpasses the capabilities of current methods. Users desire a more versatile approach that can accommodate their diverse creative intents, ranging from controlling individual subjects to manipulating the entire scene composition. We present VersaGen, a generative AI agent that enables versatile visual control in T2I synthesis. VersaGen admits four types of visual controls: i) single visual subject; ii) multiple visual subjects; iii) scene background; iv) any combination of the three above or merely no control at all. We train an adaptor upon a frozen T2I model to accommodate the visual information into the text-dominated diffusion process. We introduce three optimization strategies during the inference phase of VersaGen to improve generation results and enhance user experience. Comprehensive experiments on COCO and Sketchy validate the effectiveness and flexibility of VersaGen, as evidenced by both qualitative and quantitative results.