Semi-Supervised Adaptation of Diffusion Models for Handwritten Text Generation
This work addresses the challenge of adapting handwritten text generation models to new, unlabeled datasets, which is incremental but important for applications like training data augmentation in document analysis.
The paper tackles the problem of generating realistic handwritten text images for unseen writing styles by extending a latent diffusion model with a masked autoencoder for style conditioning and a semi-supervised training scheme, achieving improvements on the IAM and RIMES databases.
The generation of images of realistic looking, readable handwritten text is a challenging task which is referred to as handwritten text generation (HTG). Given a string and examples from a writer, the goal is to synthesize an image depicting the correctly spelled word in handwriting with the calligraphic style of the desired writer. An important application of HTG is the generation of training images in order to adapt downstream models for new data sets. With their success in natural image generation, diffusion models (DMs) have become the state-of-the-art approach in HTG. In this work, we present an extension of a latent DM for HTG to enable generation of writing styles not seen during training by learning style conditioning with a masked auto encoder. Our proposed content encoder allows for different ways of conditioning the DM on textual and calligraphic features. Additionally, we employ classifier-free guidance and explore the influence on the quality of the generated training images. For adapting the model to a new unlabeled data set, we propose a semi-supervised training scheme. We evaluate our approach on the IAM-database and use the RIMES-database to examine the generation of data not seen during training achieving improvements in this particularly promising application of DMs for HTG.