Part of speech and gramset tagging algorithms for unknown words based on morphological dictionaries of the Veps and Karelian languages
This work addresses morphological tagging challenges for low-resource languages, providing incremental improvements using existing methods on new data.
The research tackled part-of-speech and gramset tagging for unknown words in low-resource Veps and Karelian languages, achieving accuracies of 92.4% for Veps and 86.8% for Karelian in part-of-speech tagging, and 95.3% for Veps and 90.7% for Karelian in gramset tagging.
This research devoted to the low-resource Veps and Karelian languages. Algorithms for assigning part of speech tags to words and grammatical properties to words are presented in the article. These algorithms use our morphological dictionaries, where the lemma, part of speech and a set of grammatical features (gramset) are known for each word form. The algorithms are based on the analogy hypothesis that words with the same suffixes are likely to have the same inflectional models, the same part of speech and gramset. The accuracy of these algorithms were evaluated and compared. 313 thousand Vepsian and 66 thousand Karelian words were used to verify the accuracy of these algorithms. The special functions were designed to assess the quality of results of the developed algorithms. 92.4% of Vepsian words and 86.8% of Karelian words were assigned a correct part of speech by the developed algorithm. 95.3% of Vepsian words and 90.7% of Karelian words were assigned a correct gramset by our algorithm. Morphological and semantic tagging of texts, which are closely related and inseparable in our corpus processes, are described in the paper.