LDC Arabic Treebanks and Associated Corpora: Data Divisions Manual
This work provides a standardized approach for data preparation in Arabic NLP, but it is incremental as it focuses on data management rather than novel methods or results.
The paper addresses the need for consistent data divisions in Arabic treebanks to support NLP research by detailing a set of rules for creating training, development, and evaluation sets from existing corpora.
The Linguistic Data Consortium (LDC) has developed hundreds of data corpora for natural language processing (NLP) research. Among these are a number of annotated treebank corpora for Arabic. Typically, these corpora consist of a single collection of annotated documents. NLP research, however, usually requires multiple data sets for the purposes of training models, developing techniques, and final evaluation. Therefore it becomes necessary to divide the corpora used into the required data sets (divisions). This document details a set of rules that have been defined to enable consistent divisions for old and new Arabic treebanks (ATB) and related corpora.