Evaluation Discrepancy Discovery: A Sentence Compression Case-study
This exposes evaluation flaws in NLP research, which is critical for reproducibility but incremental as it focuses on a specific case-study.
The paper tackled the problem of unreliable evaluation in NLP by showing that a sentence compression system could achieve state-of-the-art results by gaming a dataset, with manual analysis revealing high metric scores did not correlate with better human-perceived outputs.
Reliable evaluation protocols are of utmost importance for reproducible NLP research. In this work, we show that sometimes neither metric nor conventional human evaluation is sufficient to draw conclusions about system performance. Using sentence compression as an example task, we demonstrate how a system can game a well-established dataset to achieve state-of-the-art results. In contrast with the results reported in previous work that showed correlation between human judgements and metric scores, our manual analysis of state-of-the-art system outputs demonstrates that high metric scores may only indicate a better fit to the data, but not better outputs, as perceived by humans.