Targeting the Benchmark: On Methodology in Current Natural Language Processing Research
This addresses a methodological issue for NLP researchers, highlighting implicit assumptions in benchmark-driven progress, and is incremental as it builds on existing critiques without proposing new methods.
The paper critiques the common pattern in NLP research where new benchmarks are introduced and quickly improved upon, often without clear justification for progress, by analyzing possible argumentations and their components.
It has become a common pattern in our field: One group introduces a language task, exemplified by a dataset, which they argue is challenging enough to serve as a benchmark. They also provide a baseline model for it, which then soon is improved upon by other groups. Often, research efforts then move on, and the pattern repeats itself. What is typically left implicit is the argumentation for why this constitutes progress, and progress towards what. In this paper, we try to step back for a moment from this pattern and work out possible argumentations and their parts.