DialectGram: Detecting Dialectal Variation at Multiple Geographic Resolutions
This addresses the need for flexible dialect analysis tools for linguists and NLP researchers, offering a novel approach that avoids retraining for different resolutions.
The paper tackles the problem of detecting dialectal variation at multiple geographic resolutions without predefined regions, proposing DialectGram, which learns dialect-sensitive word embeddings and models senses, and shows it effectively models linguistic variation on the DialectSim dataset.
Several computational models have been developed to detect and analyze dialect variation in recent years. Most of these models assume a predefined set of geographical regions over which they detect and analyze dialectal variation. However, dialect variation occurs at multiple levels of geographic resolution ranging from cities within a state, states within a country, and between countries across continents. In this work, we propose a model that enables detection of dialectal variation at multiple levels of geographic resolution obviating the need for a-priori definition of the resolution level. Our method DialectGram, learns dialect-sensitive word embeddings while being agnostic of the geographic resolution. Specifically it only requires one-time training and enables analysis of dialectal variation at a chosen resolution post-hoc -- a significant departure from prior models which need to be re-trained whenever the pre-defined set of regions changes. Furthermore, DialectGram explicitly models senses thus enabling one to estimate the proportion of each sense usage in any given region. Finally, we quantitatively evaluate our model against other baselines on a new evaluation dataset DialectSim (in English) and show that DialectGram can effectively model linguistic variation.