Using Fisher's Exact Test to Evaluate Association Measures for N-grams
This work addresses the problem of selecting reliable association measures for n-gram analysis in computational linguistics, offering incremental improvements in evaluation methods.
The study evaluated lexical association measures for n-grams using Fisher's exact test on a 4-million-word corpus, finding that simple-ll is highly effective and MI3 performs nearly as well for 3-grams, with some measures showing efficiency variations between 2-grams and 3-grams.
To determine whether some often-used lexical association measures assign high scores to n-grams that chance could have produced as frequently as observed, we used an extension of Fisher's exact test to sequences longer than two words to analyse a corpus of four million words. The results, based on the precision-recall curve and a new index called chance-corrected average precision, show that, as expected, simple-ll is extremely effective. They also show, however, that MI3 is more efficient than the other hypothesis tests-based measures and even reaches a performance level almost equal to simple-ll for 3-grams. It is additionally observed that some measures are more efficient for 3-grams than for 2-grams, while others stagnate.