OCR-VQGAN: Taming Text-within-Image Generation
This addresses a specific challenge in figure and diagram generation for fields like AI and computer vision, though it is incremental as it builds on existing VQGAN methods.
The paper tackles the problem of generating readable text within synthetic figures and diagrams by introducing OCR-VQGAN, which uses OCR pre-trained features and a text perceptual loss to preserve text fidelity and structure, achieving improved reconstruction results on a new dataset of over 100k research paper figures.
Synthetic image generation has recently experienced significant improvements in domains such as natural image or art generation. However, the problem of figure and diagram generation remains unexplored. A challenging aspect of generating figures and diagrams is effectively rendering readable texts within the images. To alleviate this problem, we present OCR-VQGAN, an image encoder, and decoder that leverages OCR pre-trained features to optimize a text perceptual loss, encouraging the architecture to preserve high-fidelity text and diagram structure. To explore our approach, we introduce the Paper2Fig100k dataset, with over 100k images of figures and texts from research papers. The figures show architecture diagrams and methodologies of articles available at arXiv.org from fields like artificial intelligence and computer vision. Figures usually include text and discrete objects, e.g., boxes in a diagram, with lines and arrows that connect them. We demonstrate the effectiveness of OCR-VQGAN by conducting several experiments on the task of figure reconstruction. Additionally, we explore the qualitative and quantitative impact of weighting different perceptual metrics in the overall loss function. We release code, models, and dataset at https://github.com/joanrod/ocr-vqgan.