InteChar: A Unified Oracle Bone Character List for Ancient Chinese Language Modeling
This addresses the problem of digitizing and modeling ancient Chinese scripts for archaeologists and NLP researchers, but it is incremental as it builds on existing methods for character encoding and corpus construction.
The paper tackles the challenge of training historical language models on ancient Chinese texts by introducing InteChar, a unified character list that integrates unencoded oracle bone characters with traditional and modern Chinese, and models trained with it on the OracleCS corpus achieve substantial improvements in historical language understanding tasks.
Constructing historical language models (LMs) plays a crucial role in aiding archaeological provenance studies and understanding ancient cultures. However, existing resources present major challenges for training effective LMs on historical texts. First, the scarcity of historical language samples renders unsupervised learning approaches based on large text corpora highly inefficient, hindering effective pre-training. Moreover, due to the considerable temporal gap and complex evolution of ancient scripts, the absence of comprehensive character encoding schemes limits the digitization and computational processing of ancient texts, particularly in early Chinese writing. To address these challenges, we introduce InteChar, a unified and extensible character list that integrates unencoded oracle bone characters with traditional and modern Chinese. InteChar enables consistent digitization and representation of historical texts, providing a foundation for robust modeling of ancient scripts. To evaluate the effectiveness of InteChar, we construct the Oracle Corpus Set (OracleCS), an ancient Chinese corpus that combines expert-annotated samples with LLM-assisted data augmentation, centered on Chinese oracle bone inscriptions. Extensive experiments show that models trained with InteChar on OracleCS achieve substantial improvements across various historical language understanding tasks, confirming the effectiveness of our approach and establishing a solid foundation for future research in ancient Chinese NLP.