Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models
This work addresses a bottleneck in text-to-image generation for users needing faster and more accurate outputs, representing an incremental improvement over existing token-based methods.
The paper tackles the suboptimal generation performance of token-based masked generative models in text-to-image generation by proposing a text-conditioned sampling framework, which improves image quality and semantic alignment with texts and reduces inference time by over 50% without modifying the original model.
Token-based masked generative models are gaining popularity for their fast inference time with parallel decoding. While recent token-based approaches achieve competitive performance to diffusion-based models, their generation performance is still suboptimal as they sample multiple tokens simultaneously without considering the dependence among them. We empirically investigate this problem and propose a learnable sampling model, Text-Conditioned Token Selection (TCTS), to select optimal tokens via localized supervision with text information. TCTS improves not only the image quality but also the semantic alignment of the generated images with the given texts. To further improve the image quality, we introduce a cohesive sampling strategy, Frequency Adaptive Sampling (FAS), to each group of tokens divided according to the self-attention maps. We validate the efficacy of TCTS combined with FAS with various generative tasks, demonstrating that it significantly outperforms the baselines in image-text alignment and image quality. Our text-conditioned sampling framework further reduces the original inference time by more than 50% without modifying the original generative model.