Sentence-level dialects identification in the greater China region
This work addresses the problem of dialect identification for linguists and NLP applications in the Greater China Region, representing an incremental improvement over existing methods.
The paper tackled the challenge of identifying Mandarin Chinese dialects across the Greater China Region by addressing inefficiencies in character-level or word-level unigram features due to ambiguity and context-dependency. The proposed approach, incorporating new word-level features like PMI-based and word alignment-based ones, showed effectiveness in evaluations on news and Wikipedia datasets.
Identifying the different varieties of the same language is more challenging than unrelated languages identification. In this paper, we propose an approach to discriminate language varieties or dialects of Mandarin Chinese for the Mainland China, Hong Kong, Taiwan, Macao, Malaysia and Singapore, a.k.a., the Greater China Region (GCR). When applied to the dialects identification of the GCR, we find that the commonly used character-level or word-level uni-gram feature is not very efficient since there exist several specific problems such as the ambiguity and context-dependent characteristic of words in the dialects of the GCR. To overcome these challenges, we use not only the general features like character-level n-gram, but also many new word-level features, including PMI-based and word alignment-based features. A series of evaluation results on both the news and open-domain dataset from Wikipedia show the effectiveness of the proposed approach.