Large-Scale Evaluation of Keyphrase Extraction Models
This work addresses the issue of unreliable benchmarking in keyphrase extraction for researchers and practitioners, providing recommendations for better evaluation practices.
The authors tackled the problem of inconsistent evaluation in keyphrase extraction by conducting a large-scale analysis of state-of-the-art models across multiple datasets, revealing that these models are often outperformed by simple baselines in some cases.
Keyphrase extraction models are usually evaluated under different, not directly comparable, experimental setups. As a result, it remains unclear how well proposed models actually perform, and how they compare to each other. In this work, we address this issue by presenting a systematic large-scale analysis of state-of-the-art keyphrase extraction models involving multiple benchmark datasets from various sources and domains. Our main results reveal that state-of-the-art models are in fact still challenged by simple baselines on some datasets. We also present new insights about the impact of using author- or reader-assigned keyphrases as a proxy for gold standard, and give recommendations for strong baselines and reliable benchmark datasets.