Benchmarking Large Language Models for Handwritten Text Recognition
This work addresses the problem of reducing annotation needs in handwritten text recognition for researchers and practitioners, but it is incremental as it benchmarks existing models without introducing new methods.
The study benchmarked large language models for handwritten text recognition, finding that proprietary models like Claude 3.5 Sonnet outperformed open-source ones in zero-shot settings, with no consistent advantage over traditional Transkribus models and limited autonomous error correction ability.
Traditional machine learning models for Handwritten Text Recognition (HTR) rely on supervised training, requiring extensive manual annotations, and often produce errors due to the separation between layout and text processing. In contrast, Multimodal Large Language Models (MLLMs) offer a general approach to recognizing diverse handwriting styles without the need for model-specific training. The study benchmarks various proprietary and open-source LLMs against Transkribus models, evaluating their performance on both modern and historical datasets written in English, French, German, and Italian. In addition, emphasis is placed on testing the models' ability to autonomously correct previously generated outputs. Findings indicate that proprietary models, especially Claude 3.5 Sonnet, outperform open-source alternatives in zero-shot settings. MLLMs achieve excellent results in recognizing modern handwriting and exhibit a preference for the English language due to their pre-training dataset composition. Comparisons with Transkribus show no consistent advantage for either approach. Moreover, LLMs demonstrate limited ability to autonomously correct errors in zero-shot transcriptions.