DiffusionPen: Towards Controlling the Style of Handwritten Text Generation
This work addresses the problem of generating diverse and realistic handwritten text for applications like data augmentation in handwriting recognition, though it is incremental as it builds on existing diffusion models.
The paper tackles the challenge of generating handwritten text conditioned on both content and style by introducing DiffusionPen, a 5-shot approach based on Latent Diffusion Models that captures textual and stylistic characteristics for seen and unseen words and styles, generating realistic samples and outperforming existing methods in experiments on the IAM database.
Handwritten Text Generation (HTG) conditioned on text and style is a challenging task due to the variability of inter-user characteristics and the unlimited combinations of characters that form new words unseen during training. Diffusion Models have recently shown promising results in HTG but still remain under-explored. We present DiffusionPen (DiffPen), a 5-shot style handwritten text generation approach based on Latent Diffusion Models. By utilizing a hybrid style extractor that combines metric learning and classification, our approach manages to capture both textual and stylistic characteristics of seen and unseen words and styles, generating realistic handwritten samples. Moreover, we explore several variation strategies of the data with multi-style mixtures and noisy embeddings, enhancing the robustness and diversity of the generated data. Extensive experiments using IAM offline handwriting database show that our method outperforms existing methods qualitatively and quantitatively, and its additional generated data can improve the performance of Handwriting Text Recognition (HTR) systems. The code is available at: https://github.com/koninik/DiffusionPen.