PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generation
This work addresses the lack of specialized models for Khmer language processing, which is an incremental improvement over existing multilingual approaches.
The authors tackled the problem of natural language generation for Khmer by developing PrahokBART, a pre-trained sequence-to-sequence model trained from scratch, which outperformed the multilingual mBART50 model on tasks like machine translation, text summarization, and headline generation.
This work introduces {\it PrahokBART}, a compact pre-trained sequence-to-sequence model trained from scratch for Khmer using carefully curated Khmer and English corpora. We focus on improving the pre-training corpus quality and addressing the linguistic issues of Khmer, which are ignored in existing multilingual models, by incorporating linguistic components such as word segmentation and normalization. We evaluate PrahokBART on three generative tasks: machine translation, text summarization, and headline generation, where our results demonstrate that it outperforms mBART50, a strong multilingual pre-trained model. Additionally, our analysis provides insights into the impact of each linguistic module and evaluates how effectively our model handles space during text generation, which is crucial for the naturalness of texts in Khmer.