Text-to-Image Generation Grounded by Fine-Grained User Attention
This addresses the challenge of precise visual grounding in text-to-image generation for applications requiring detailed image synthesis, though it is incremental as it builds on retrieval and segmentation techniques.
The paper tackles the problem of generating images from text by using fine-grained user attention from mouse traces to ground descriptions, resulting in a model that outperforms existing text-to-image methods in photo-realism and description matching.
Localized Narratives is a dataset with detailed natural language descriptions of images paired with mouse traces that provide a sparse, fine-grained visual grounding for phrases. We propose TReCS, a sequential model that exploits this grounding to generate images. TReCS uses descriptions to retrieve segmentation masks and predict object labels aligned with mouse traces. These alignments are used to select and position masks to generate a fully covered segmentation canvas; the final image is produced by a segmentation-to-image generator using this canvas. This multi-step, retrieval-based approach outperforms existing direct text-to-image generation models on both automatic metrics and human evaluations: overall, its generated images are more photo-realistic and better match descriptions.