Social Good or Scientific Curiosity? Uncovering the Research Framing Behind NLP Artefacts
This work addresses the issue of aligning NLP research with practical applications by uncovering trends like vague goals and shifts in focus, which is incremental as it builds on prior manual analyses.
The paper tackled the problem of clarifying research framings in NLP artefacts by automating the analysis of key elements like stakeholders and intended uses, achieving consistent improvements over strong LLM baselines in domains such as automated fact-checking and hate speech detection.
Clarifying the research framing of NLP artefacts (e.g., models, datasets, etc.) is crucial to aligning research with practical applications. Recent studies manually analyzed NLP research across domains, showing that few papers explicitly identify key stakeholders, intended uses, or appropriate contexts. In this work, we propose to automate this analysis, developing a three-component system that infers research framings by first extracting key elements (means, ends, stakeholders), then linking them through interpretable rules and contextual reasoning. We evaluate our approach on two domains: automated fact-checking using an existing dataset, and hate speech detection for which we annotate a new dataset-achieving consistent improvements over strong LLM baselines. Finally, we apply our system to recent automated fact-checking papers and uncover three notable trends: a rise in vague or underspecified research goals, increased emphasis on scientific exploration over application, and a shift toward supporting human fact-checkers rather than pursuing full automation.