Comparing Open Arabic Named Entity Recognition Tools
This work provides a practical evaluation for researchers and practitioners using Arabic NER tools, but it is incremental as it focuses on comparing existing methods without introducing new models.
The paper compared three open Arabic NER tools (CAMeL, Hatmi, Stanza) on a manually annotated corpus, finding Hatmi had the highest F1 scores and CAMeL the highest precision for some entities, and showed that merging their results achieved the highest overall F1 scores while voting optimized precision, with these tradeoffs confirmed on a larger COVID-19 dataset.
The main objective of this paper is to compare and evaluate the performances of three open Arabic NER tools: CAMeL, Hatmi, and Stanza. We collected a corpus consisting of 30 articles written in MSA and manually annotated all the entities of the person, organization, and location types at the article (document) level. Our results suggest a similarity between Stanza and Hatmi with the latter receiving the highest F1 score for the three entity types. However, CAMeL achieved the highest precision values for names of people and organizations. Following this, we implemented a "merge" method that combined the results from the three tools and a "vote" method that tagged named entities only when two of the three identified them as entities. Our results showed that merging achieved the highest overall F1 scores. Moreover, merging had the highest recall values while voting had the highest precision values for the three entity types. This indicates that merging is more suitable when recall is desired, while voting is optimal when precision is required. Finally, we collected a corpus of 21,635 articles related to COVID-19 and applied the merge and vote methods. Our analysis demonstrates the tradeoff between precision and recall for the two methods.