Evaluating LLMs for Historical Document OCR: A Methodological Framework for Digital Humanities
This provides digital humanities practitioners with a framework for model selection and quality assessment in historical corpus digitization, addressing a specific gap in evaluation methods.
The paper tackles the problem of evaluating Large Language Models for historical document OCR in digital humanities, introducing a methodological framework with novel metrics like Historical Character Preservation Rate and Archaic Insertion Rate. They evaluated 12 multimodal LLMs on 18th-century Russian Civil font texts, finding that Gemini and Qwen models outperform traditional OCR but exhibit over-historicization errors, with post-OCR correction degrading performance.
Digital humanities scholars increasingly use Large Language Models for historical document digitization, yet lack appropriate evaluation frameworks for LLM-based OCR. Traditional metrics fail to capture temporal biases and period-specific errors crucial for historical corpus creation. We present an evaluation methodology for LLM-based historical OCR, addressing contamination risks and systematic biases in diplomatic transcription. Using 18th-century Russian Civil font texts, we introduce novel metrics including Historical Character Preservation Rate (HCPR) and Archaic Insertion Rate (AIR), alongside protocols for contamination control and stability testing. We evaluate 12 multimodal LLMs, finding that Gemini and Qwen models outperform traditional OCR while exhibiting over-historicization: inserting archaic characters from incorrect historical periods. Post-OCR correction degrades rather than improves performance. Our methodology provides digital humanities practitioners with guidelines for model selection and quality assessment in historical corpus digitization.