INDIC DIALECT: A Multi Task Benchmark to Evaluate and Translate in Indian Language Dialects
This addresses the problem of limited NLP resources for Indian dialect speakers, though it is incremental as it builds on existing datasets and methods for low-resource languages.
The paper tackles the under-representation of low-resource Indian dialects in NLP by introducing INDIC-DIALECT, a human-curated parallel corpus of 13k sentence pairs across 11 dialects and 2 languages, and constructs a multi-task benchmark. Results show that fine-tuned transformer models improve dialect classification F1 from 19.6% to 89.8%, and hybrid AI models achieve BLEU scores up to 61.32 for dialect-to-language translation.
Recent NLP advances focus primarily on standardized languages, leaving most low-resource dialects under-served especially in Indian scenarios. In India, the issue is particularly important: despite Hindi being the third most spoken language globally (over 600 million speakers), its numerous dialects remain underrepresented. The situation is similar for Odia, which has around 45 million speakers. While some datasets exist which contain standard Hindi and Odia languages, their regional dialects have almost no web presence. We introduce INDIC-DIALECT, a human-curated parallel corpus of 13k sentence pairs spanning 11 dialects and 2 languages: Hindi and Odia. Using this corpus, we construct a multi-task benchmark with three tasks: dialect classification, multiple-choice question (MCQ) answering, and machine translation (MT). Our experiments show that LLMs like GPT-4o and Gemini 2.5 perform poorly on the classification task. While fine-tuned transformer based models pretrained on Indian languages substantially improve performance e.g., improving F1 from 19.6\% to 89.8\% on dialect classification. For dialect to language translation, we find that hybrid AI model achieves highest BLEU score of 61.32 compared to the baseline score of 23.36. Interestingly, due to complexity in generating dialect sentences, we observe that for language to dialect translation the ``rule-based followed by AI" approach achieves best BLEU score of 48.44 compared to the baseline score of 27.59. INDIC-DIALECT thus is a new benchmark for dialect-aware Indic NLP, and we plan to release it as open source to support further work on low-resource Indian dialects.