Multimodal LLMs for OCR, OCR Post-Correction, and Named Entity Recognition in Historical Documents
This work addresses the challenge of historical data collection and document transcription for researchers, offering a novel approach with potential paradigm-shifting impact.
The study tackled the problem of transcribing and extracting information from historical documents by applying multimodal Large Language Models (mLLMs) to OCR, OCR post-correction, and NER tasks on German city directories from 1754-1870, finding that mLLMs significantly outperformed conventional OCR models and achieved highly accurate transcriptions (<1% CER) without fine-tuning.
We explore how multimodal Large Language Models (mLLMs) can help researchers transcribe historical documents, extract relevant historical information, and construct datasets from historical sources. Specifically, we investigate the capabilities of mLLMs in performing (1) Optical Character Recognition (OCR), (2) OCR Post-Correction, and (3) Named Entity Recognition (NER) tasks on a set of city directories published in German between 1754 and 1870. First, we benchmark the off-the-shelf transcription accuracy of both mLLMs and conventional OCR models. We find that the best-performing mLLM model significantly outperforms conventional state-of-the-art OCR models and other frontier mLLMs. Second, we are the first to introduce multimodal post-correction of OCR output using mLLMs. We find that this novel approach leads to a drastic improvement in transcription accuracy and consistently produces highly accurate transcriptions (<1% CER), without any image pre-processing or model fine-tuning. Third, we demonstrate that mLLMs can efficiently recognize entities in transcriptions of historical documents and parse them into structured dataset formats. Our findings provide early evidence for the long-term potential of mLLMs to introduce a paradigm shift in the approaches to historical data collection and document transcription.