Keyphrase Based Evaluation of Automatic Text Summarization
This is an incremental improvement for researchers in natural language processing, specifically for evaluating Arabic text summaries.
The study tackled the problem of inaccurate matching in automatic summary evaluation systems by introducing KpEval, a keyphrase-based evaluator, which showed strong correlations with existing systems, such as a Pearson coefficient of 0.8840 with AutoSummENG MeMoG.
The development of methods to deal with the informative contents of the text units in the matching process is a major challenge in automatic summary evaluation systems that use fixed n-gram matching. The limitation causes inaccurate matching between units in a peer and reference summaries. The present study introduces a new Keyphrase based Summary Evaluator KpEval for evaluating automatic summaries. The KpEval relies on the keyphrases since they convey the most important concepts of a text. In the evaluation process, the keyphrases are used in their lemma form as the matching text unit. The system was applied to evaluate different summaries of Arabic multi-document data set presented at TAC2011. The results showed that the new evaluation technique correlates well with the known evaluation systems: Rouge1, Rouge2, RougeSU4, and AutoSummENG MeMoG. KpEval has the strongest correlation with AutoSummENG MeMoG, Pearson and spearman correlation coefficient measures are 0.8840, 0.9667 respectively.