Learning to Recognize Dialect Features
This addresses the challenge of building inclusive NLP systems by enabling dialect feature detection with limited data, though it is incremental as it builds on existing multitask learning and pretrained transformers.
The paper tackles the problem of recognizing dialect features in NLP by introducing dialect feature detection and training models on minimal pairs, achieving high accuracy on 22 Indian English features and showing that minimal pairs can be as effective as thousands of labeled examples.
Building NLP systems that serve everyone requires accounting for dialect differences. But dialects are not monolithic entities: rather, distinctions between and within dialects are captured by the presence, absence, and frequency of dozens of dialect features in speech and text, such as the deletion of the copula in "He {} running". In this paper, we introduce the task of dialect feature detection, and present two multitask learning approaches, both based on pretrained transformers. For most dialects, large-scale annotated corpora for these features are unavailable, making it difficult to train recognizers. We train our models on a small number of minimal pairs, building on how linguists typically define dialect features. Evaluation on a test set of 22 dialect features of Indian English demonstrates that these models learn to recognize many features with high accuracy, and that a few minimal pairs can be as effective for training as thousands of labeled examples. We also demonstrate the downstream applicability of dialect feature detection both as a measure of dialect density and as a dialect classifier.