SemEval-2013 Task 4: Free Paraphrases of Noun Compounds
This addresses the challenge of semantic interpretation for natural language processing researchers, but it is incremental as it builds on existing paraphrase tasks.
The paper tackled the problem of capturing meaning in English noun compounds by developing a task for systems to produce ranked free-form paraphrases, with three participating systems beating a baseline on one of two evaluation measures but not both, indicating difficulty.
In this paper, we describe SemEval-2013 Task 4: the definition, the data, the evaluation and the results. The task is to capture some of the meaning of English noun compounds via paraphrasing. Given a two-word noun compound, the participating system is asked to produce an explicitly ranked list of its free-form paraphrases. The list is automatically compared and evaluated against a similarly ranked list of paraphrases proposed by human annotators, recruited and managed through Amazon's Mechanical Turk. The comparison of raw paraphrases is sensitive to syntactic and morphological variation. The "gold" ranking is based on the relative popularity of paraphrases among annotators. To make the ranking more reliable, highly similar paraphrases are grouped, so as to downplay superficial differences in syntax and morphology. Three systems participated in the task. They all beat a simple baseline on one of the two evaluation measures, but not on both measures. This shows that the task is difficult.