Lemmatization as a Classification Task: Results from Arabic across Multiple Genres
This addresses lemmatization challenges for Arabic NLP, which is crucial for morphologically rich languages, though it appears incremental with new methods and datasets.
The paper tackles Arabic lemmatization by framing it as classification into a Lemma-POS-Gloss tagset using machine translation and semantic clustering, and introduces a new test set covering diverse genres. Results show classification and clustering methods set new benchmarks, with character-level sequence-to-sequence models performing competitively but limited to lemma prediction and prone to hallucinations.
Lemmatization is crucial for NLP tasks in morphologically rich languages with ambiguous orthography like Arabic, but existing tools face challenges due to inconsistent standards and limited genre coverage. This paper introduces two novel approaches that frame lemmatization as classification into a Lemma-POS-Gloss (LPG) tagset, leveraging machine translation and semantic clustering. We also present a new Arabic lemmatization test set covering diverse genres, standardized alongside existing datasets. We evaluate character level sequence-to-sequence models, which perform competitively and offer complementary value, but are limited to lemma prediction (not LPG) and prone to hallucinating implausible forms. Our results show that classification and clustering yield more robust, interpretable outputs, setting new benchmarks for Arabic lemmatization.