Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research
It addresses a methodological problem for NLP researchers by questioning incremental progress and advocating for foundational clarity.
The paper critiques the assumption that introducing new tasks or improving state-of-the-art metrics in NLP inherently constitutes progress, and it aims to clarify the concepts of language tasks and games to guide research toward modeling general language competence.
"This paper introduces a new task and a new dataset", "we improve the state of the art in X by Y" -- it is rare to find a current natural language processing paper (or AI paper more generally) that does not contain such statements. What is mostly left implicit, however, is the assumption that this necessarily constitutes progress, and what it constitutes progress towards. Here, we make more precise the normally impressionistically used notions of language task and language game and ask how a research programme built on these might make progress towards the goal of modelling general language competence.