ArtAug: Enhancing Text-to-Image Generation through Synthesis-Understanding Interaction
This addresses the challenge of producing aesthetically pleasing images in text-to-image generation for users in creative and AI applications, though it appears incremental as it builds on existing models with a novel interaction approach.
The paper tackles the problem of enhancing text-to-image generation by proposing ArtAug, a method that uses interactions with image understanding models to provide fine-grained suggestions for aesthetic improvements, such as adjusting exposure and adding atmospheric effects, resulting in enhanced generative capabilities without extra computational costs, as demonstrated by various evaluation metrics.
The emergence of diffusion models has significantly advanced image synthesis. The recent studies of model interaction and self-corrective reasoning approach in large language models offer new insights for enhancing text-to-image models. Inspired by these studies, we propose a novel method called ArtAug for enhancing text-to-image models in this paper. To the best of our knowledge, ArtAug is the first one that improves image synthesis models via model interactions with understanding models. In the interactions, we leverage human preferences implicitly learned by image understanding models to provide fine-grained suggestions for image synthesis models. The interactions can modify the image content to make it aesthetically pleasing, such as adjusting exposure, changing shooting angles, and adding atmospheric effects. The enhancements brought by the interaction are iteratively fused into the synthesis model itself through an additional enhancement module. This enables the synthesis model to directly produce aesthetically pleasing images without any extra computational cost. In the experiments, we train the ArtAug enhancement module on existing text-to-image models. Various evaluation metrics consistently demonstrate that ArtAug enhances the generative capabilities of text-to-image models without incurring additional computational costs. The source code and models will be released publicly.