IruMozhi: Automatically classifying diglossia in Tamil
This addresses the under-support of Spoken Tamil in NLP systems, which is an incremental step for Tamil language processing.
The paper tackles the problem of diglossia in Tamil by releasing IruMozhi, a human-annotated dataset of parallel text in Literary and Spoken Tamil, and trains classifiers to identify the variety, using them to assess pretraining data availability and audit existing datasets.
Tamil, a Dravidian language of South Asia, is a highly diglossic language with two very different registers in everyday use: Literary Tamil (preferred in writing and formal communication) and Spoken Tamil (confined to speech and informal media). Spoken Tamil is under-supported in modern NLP systems. In this paper, we release IruMozhi, a human-annotated dataset of parallel text in Literary and Spoken Tamil. We train classifiers on the task of identifying which variety a text belongs to. We use these models to gauge the availability of pretraining data in Spoken Tamil, to audit the composition of existing labelled datasets for Tamil, and to encourage future work on the variety.