Rethinking HTG Evaluation: Bridging Generation and Recognition
This work addresses the need for standardized evaluation protocols in handwriting generation, which is important for researchers and practitioners in document analysis and recognition, though it is incremental as it refines existing evaluation approaches.
The paper tackles the problem of evaluating generative models for handwriting generation (HTG) by introducing three tailored metrics (HTG_HTR, HTG_style, HTG_OOV) that emphasize style, content, and diversity, showing they provide richer information than widely used metrics like FID in experiments on the IAM database.
The evaluation of generative models for natural image tasks has been extensively studied. Similar protocols and metrics are used in cases with unique particularities, such as Handwriting Generation, even if they might not be completely appropriate. In this work, we introduce three measures tailored for HTG evaluation, $ \text{HTG}_{\text{HTR}} $, $ \text{HTG}_{\text{style}} $, and $ \text{HTG}_{\text{OOV}} $, and argue that they are more expedient to evaluate the quality of generated handwritten images. The metrics rely on the recognition error/accuracy of Handwriting Text Recognition and Writer Identification models and emphasize writing style, textual content, and diversity as the main aspects that adhere to the content of handwritten images. We conduct comprehensive experiments on the IAM handwriting database, showcasing that widely used metrics such as FID fail to properly quantify the diversity and the practical utility of generated handwriting samples. Our findings show that our metrics are richer in information and underscore the necessity of standardized evaluation protocols in HTG. The proposed metrics provide a more robust and informative protocol for assessing HTG quality, contributing to improved performance in HTR. Code for the evaluation protocol is available at: https://github.com/koninik/HTG_evaluation.