DPRK-BERT: The Supreme Language Model
This addresses the problem of low-resource language modeling for DPRK language researchers, though it is incremental as it adapts existing methods to new data.
The authors tackled the lack of deep language models for the DPRK language by creating DPRK-BERT, the first such model, which shows significant improvements on two DPRK datasets and better cross-lingual generalization with ROK language models.
Deep language models have achieved remarkable success in the NLP domain. The standard way to train a deep language model is to employ unsupervised learning from scratch on a large unlabeled corpus. However, such large corpora are only available for widely-adopted and high-resource languages and domains. This study presents the first deep language model, DPRK-BERT, for the DPRK language. We achieve this by compiling the first unlabeled corpus for the DPRK language and fine-tuning a preexisting the ROK language model. We compare the proposed model with existing approaches and show significant improvements on two DPRK datasets. We also present a cross-lingual version of this model which yields better generalization across the two Korean languages. Finally, we provide various NLP tools related to the DPRK language that would foster future research.