Fine-Tashkeel: Finetuning Byte-Level Models for Accurate Arabic Text Diacritization
This improves diacritization accuracy for Arabic language processing, though it is incremental as it applies existing finetuning methods to a specific task.
The paper tackles Arabic text diacritization by finetuning pre-trained multilingual models (ByT5) to predict missing diacritics, achieving state-of-the-art results with a 40% reduction in WER and minimal training.
Most of previous work on learning diacritization of the Arabic language relied on training models from scratch. In this paper, we investigate how to leverage pre-trained language models to learn diacritization. We finetune token-free pre-trained multilingual models (ByT5) to learn to predict and insert missing diacritics in Arabic text, a complex task that requires understanding the sentence semantics and the morphological structure of the tokens. We show that we can achieve state-of-the-art on the diacritization task with minimal amount of training and no feature engineering, reducing WER by 40%. We release our finetuned models for the greater benefit of the researchers in the community.