What's the Meaning of Superhuman Performance in Today's NLU?
This addresses the problem of misleading claims in NLP for researchers and practitioners, highlighting incremental improvements in benchmark design.
The paper critically examines claims of superhuman performance by pretrained language models on NLP benchmarks, showing that these benchmarks have serious limitations affecting human-model comparisons and providing recommendations for fairer evaluation.
In the last five years, there has been a significant focus in Natural Language Processing (NLP) on developing larger Pretrained Language Models (PLMs) and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities in language understanding, reasoning, and reading comprehension. These PLMs have achieved impressive results on these benchmarks, even surpassing human performance in some cases. This has led to claims of superhuman capabilities and the provocative idea that certain tasks have been solved. In this position paper, we take a critical look at these claims and ask whether PLMs truly have superhuman abilities and what the current benchmarks are really evaluating. We show that these benchmarks have serious limitations affecting the comparison between humans and PLMs and provide recommendations for fairer and more transparent benchmarks.