Two Spelling Normalization Approaches Based on Large Language Models
This work addresses spelling normalization for scholars in the humanities, but it is incremental as it compares existing methods rather than introducing a breakthrough.
The study tackled spelling normalization in historical documents by proposing two large language model-based approaches, one unsupervised and one trained for machine translation, and found that statistical machine translation performed best across multiple datasets.
The absence of standardized spelling conventions and the organic evolution of human language present an inherent linguistic challenge within historical documents, a longstanding concern for scholars in the humanities. Addressing this issue, spelling normalization endeavors to align a document's orthography with contemporary standards. In this study, we propose two new approaches based on large language models: one of which has been trained without a supervised training, and a second one which has been trained for machine translation. Our evaluation spans multiple datasets encompassing diverse languages and historical periods, leading us to the conclusion that while both of them yielded encouraging results, statistical machine translation still seems to be the most suitable technology for this task.