LaoPLM: Pre-trained Language Models for Lao
This work addresses the resource scarcity problem for Lao language NLP researchers, though it is incremental as it applies existing methods to a new language.
The authors tackled the under-representation of Lao in NLP by constructing a text classification dataset and developing the first transformer-based pre-trained language models for Lao, which demonstrated effectiveness in part-of-speech tagging and text classification tasks.
Trained on the large corpus, pre-trained language models (PLMs) can capture different levels of concepts in context and hence generate universal language representations. They can benefit multiple downstream natural language processing (NLP) tasks. Although PTMs have been widely used in most NLP applications, especially for high-resource languages such as English, it is under-represented in Lao NLP research. Previous work on Lao has been hampered by the lack of annotated datasets and the sparsity of language resources. In this work, we construct a text classification dataset to alleviate the resource-scare situation of the Lao language. We additionally present the first transformer-based PTMs for Lao with four versions: BERT-small, BERT-base, ELECTRA-small and ELECTRA-base, and evaluate it over two downstream tasks: part-of-speech tagging and text classification. Experiments demonstrate the effectiveness of our Lao models. We will release our models and datasets to the community, hoping to facilitate the future development of Lao NLP applications.