Research Borderlands: Analysing Writing Across Research Cultures
This work addresses the issue of cultural competence in language technologies for researchers and developers, though it is incremental as it builds on existing human-centered approaches in a specific domain.
The paper tackled the problem of measuring cultural competence in language technologies by focusing on research cultures and adapting writing across them, resulting in a framework that reveals latent cultural norms in human-written papers and highlights LLMs' tendency to homogenize writing.
Improving cultural competence of language technologies is important. However most recent works rarely engage with the communities they study, and instead rely on synthetic setups and imperfect proxies of culture. In this work, we take a human-centered approach to discover and measure language-based cultural norms, and cultural competence of LLMs. We focus on a single kind of culture, research cultures, and a single task, adapting writing across research cultures. Through a set of interviews with interdisciplinary researchers, who are experts at moving between cultures, we create a framework of structural, stylistic, rhetorical, and citational norms that vary across research cultures. We operationalise these features with a suite of computational metrics and use them for (a) surfacing latent cultural norms in human-written research papers at scale; and (b) highlighting the lack of cultural competence of LLMs, and their tendency to homogenise writing. Overall, our work illustrates the efficacy of a human-centered approach to measuring cultural norms in human-written and LLM-generated texts.