Building and curating conversational corpora for diversity-aware language science and technology
This work addresses the need for linguistically diverse conversational data to enhance language science and technology, though it is incremental as it builds on existing resources and methods.
The authors tackled the problem of building and curating conversational corpora across diverse languages, resulting in a pipeline and guidelines that cover 67 languages and varieties, with case studies demonstrating utility for understanding human interaction and improving ASR solutions.
We present an analysis pipeline and best practice guidelines for building and curating corpora of everyday conversation in diverse languages. Surveying language documentation corpora and other resources that cover 67 languages and varieties from 28 phyla, we describe the compilation and curation process, specify minimal properties of a unified format for interactional data, and develop methods for quality control that take into account turn-taking and timing. Two case studies show the broad utility of conversational data for (i) charting human interactional infrastructure and (ii) tracing challenges and opportunities for current ASR solutions. Linguistically diverse conversational corpora can provide new insights for the language sciences and stronger empirical foundations for language technology.