Associative Measures and Multi-word Unit Extraction in Turkish
This work addresses the problem of multi-word unit extraction for Turkish language processing, but it is incremental as it applies existing methods to a new dataset.
The paper tested 12 associative measures from Text-NSP on a 10-million-word Turkish corpus to evaluate their effectiveness in extracting multi-word units, focusing on optimizing corpus data and assessing rankings for linguistic relevance.
Associative measures are "mathematical formulas determining the strength of association between two or more words based on their occurrences and cooccurrences in a text corpus" (Pecina, 2010, p. 138). The purpose of this paper is to test the 12 associative measures that Text-NSP (Banerjee & Pedersen, 2003) contains on a 10-million-word subcorpus of Turkish National Corpus (TNC) (Aksan et.al., 2012). A statistical comparison of those measures is out of the scope of the study, and the measures will be evaluated according to the linguistic relevance of the rankings they provide. The focus of the study is basically on optimizing the corpus data, before applying the measures and then, evaluating the rankings produced by these measures as a whole, not on the linguistic relevance of individual n-grams. The findings include intra-linguistically relevant associative measures for a comma delimited, sentence splitted, lower-cased, well-balanced, representative, 10-million-word corpus of Turkish.