KoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding
This provides a more efficient and effective model for Korean NLP applications, though it is incremental as it adapts an existing method to a new language.
The authors tackled the lack of a pretrained ALBERT model for Korean by developing KoreALBERT, a monolingual model that outperforms BERT on 6 NLU tasks despite having fewer parameters.
A Lite BERT (ALBERT) has been introduced to scale up deep bidirectional representation learning for natural languages. Due to the lack of pretrained ALBERT models for Korean language, the best available practice is the multilingual model or resorting back to the any other BERT-based model. In this paper, we develop and pretrain KoreALBERT, a monolingual ALBERT model specifically for Korean language understanding. We introduce a new training objective, namely Word Order Prediction (WOP), and use alongside the existing MLM and SOP criteria to the same architecture and model parameters. Despite having significantly fewer model parameters (thus, quicker to train), our pretrained KoreALBERT outperforms its BERT counterpart on 6 different NLU tasks. Consistent with the empirical results in English by Lan et al., KoreALBERT seems to improve downstream task performance involving multi-sentence encoding for Korean language. The pretrained KoreALBERT is publicly available to encourage research and application development for Korean NLP.