MANorm: A Normalization Dictionary for Moroccan Arabic Dialect Written in Latin Script
This work addresses the challenge of processing user-generated text in Moroccan Arabic dialect for NLP tasks, but it is incremental as it applies existing methods to a specific domain.
The authors tackled the problem of normalizing multiple transliterations of Moroccan Arabic dialect words in Latin script to a single canonical form, using word embeddings from YouTube comments and a dialect dictionary to build MANorm, which demonstrated efficiency in normalization experiments.
Social media user-generated text is actually the main resource for many NLP tasks. This text however, does not follow the standard rules of writing. Moreover, the use of dialect such as Moroccan Arabic in written communications increases further NLP tasks complexity. A dialect is a verbal language that does not have a standard orthography, which leads users to improvise spelling while writing. Thus, for the same word we can find multiple forms of transliterations. Subsequently, it is mandatory to normalize these different transliterations to one canonical word form. To reach this goal, we have exploited the powerfulness of word embedding models generated with a corpus of YouTube comments. Besides, using a Moroccan Arabic dialect dictionary that provides the canonical forms, we have built a normalization dictionary that we refer to as MANorm. We have conducted several experiments to demonstrate the efficiency of MANorm, which have shown its usefulness in dialect normalization.