Poetry in Pixels: Prompt Tuning for Poem Image Generation via Diffusion Models
This work addresses the problem of text-to-image generation for literary works like poetry, offering a domain-specific solution that is incremental in adapting existing methods.
The authors tackled the challenge of generating images from poetry, which often has non-literal meanings, by proposing the PoemToPixel framework with prompt tuning and the PoeKey algorithm to extract key elements from poems, resulting in images that align with poetic content as validated on a new dataset.
The task of text-to-image generation has encountered significant challenges when applied to literary works, especially poetry. Poems are a distinct form of literature, with meanings that frequently transcend beyond the literal words. To address this shortcoming, we propose a PoemToPixel framework designed to generate images that visually represent the inherent meanings of poems. Our approach incorporates the concept of prompt tuning in our image generation framework to ensure that the resulting images closely align with the poetic content. In addition, we propose the PoeKey algorithm, which extracts three key elements in the form of emotions, visual elements, and themes from poems to form instructions which are subsequently provided to a diffusion model for generating corresponding images. Furthermore, to expand the diversity of the poetry dataset across different genres and ages, we introduce MiniPo, a novel multimodal dataset comprising 1001 children's poems and images. Leveraging this dataset alongside PoemSum, we conducted both quantitative and qualitative evaluations of image generation using our PoemToPixel framework. This paper demonstrates the effectiveness of our approach and offers a fresh perspective on generating images from literary sources.