On General Language Understanding
This work tackles the challenge of defining and measuring language understanding in NLP, which is crucial for improving model evaluation and ethical practices, though it is more conceptual than incremental.
The paper addresses the problem of evaluating language understanding in NLP by proposing a model that grounds measurement adequacy, highlighting that language understanding is multifaceted and influenced by situational characteristics and ethical considerations.
Natural Language Processing prides itself to be an empirically-minded, if not outright empiricist field, and yet lately it seems to get itself into essentialist debates on issues of meaning and measurement ("Do Large Language Models Understand Language, And If So, How Much?"). This is not by accident: Here, as everywhere, the evidence underspecifies the understanding. As a remedy, this paper sketches the outlines of a model of understanding, which can ground questions of the adequacy of current methods of measurement of model quality. The paper makes three claims: A) That different language use situation types have different characteristics, B) That language understanding is a multifaceted phenomenon, bringing together individualistic and social processes, and C) That the choice of Understanding Indicator marks the limits of benchmarking, and the beginnings of considerations of the ethics of NLP use.