What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP
This analysis provides insights for the NLP community on evaluation trends, but it is incremental as it synthesizes existing data without introducing new methods or benchmarks.
The paper systematically analyzes SemEval evaluation campaigns in NLP to identify patterns in task types, metrics, architectures, participation, and citations over time, aiming to clarify what is being evaluated, but it does not report concrete numerical results or specific findings.
SemEval is the primary venue in the NLP community for the proposal of new challenges and for the systematic empirical evaluation of NLP systems. This paper provides a systematic quantitative analysis of SemEval aiming to evidence the patterns of the contributions behind SemEval. By understanding the distribution of task types, metrics, architectures, participation and citations over time we aim to answer the question on what is being evaluated by SemEval.