Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP
This work addresses the need for better evaluation practices in NLP to align with models that reflect diverse perspectives, particularly for marginalized voices, though it is incremental as a position paper proposing desiderata.
The paper tackles the problem of evaluating subjectivity in NLP models by proposing seven desiderata for subjectivity-sensitive evaluation, and it finds that current practices understudy key aspects like ambiguous vs. polyphonic input and user expression of subjectivity, based on a scan of 60 papers.
Subjective judgments are part of several NLP datasets and recent work is increasingly prioritizing models whose outputs reflect this diversity of perspectives. Such responses allow us to shed light on minority voices, which are frequently marginalized or obscured by dominant perspectives. It remains a question whether our evaluation practices align with these models' objectives. This position paper proposes seven evaluation desiderata for subjectivity-sensitive models, rooted in how subjectivity is represented in NLP data and models. The desiderata are constructed in a top-down approach, keeping in mind the user-centric impact of such models. We scan the experimental setup of 60 papers and show that various aspects of subjectivity are still understudied: the distinction between ambiguous and polyphonic input, whether subjectivity is effectively expressed to the user, and a lack of interplay between different desiderata, amongst other gaps.