ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models
This work addresses the challenge of generating text-rich images for applications like design and advertising, representing an incremental improvement over existing diffusion models.
The paper tackles the problem of diffusion models generating inaccurate text in images by introducing ARTIST, a framework that uses a disentangled textual diffusion model and large language models, resulting in up to 15% improvement in metrics on the MARIO-Eval benchmark.
Diffusion models have demonstrated exceptional capabilities in generating a broad spectrum of visual content, yet their proficiency in rendering text is still limited: they often generate inaccurate characters or words that fail to blend well with the underlying image. To address these shortcomings, we introduce a novel framework named, ARTIST, which incorporates a dedicated textual diffusion model to focus on the learning of text structures specifically. Initially, we pretrain this textual model to capture the intricacies of text representation. Subsequently, we finetune a visual diffusion model, enabling it to assimilate textual structure information from the pretrained textual model. This disentangled architecture design and training strategy significantly enhance the text rendering ability of the diffusion models for text-rich image generation. Additionally, we leverage the capabilities of pretrained large language models to interpret user intentions better, contributing to improved generation quality. Empirical results on the MARIO-Eval benchmark underscore the effectiveness of the proposed method, showing an improvement of up to 15% in various metrics.