MoVoC: Morphology-Aware Subword Construction for Geez Script Languages
This work addresses tokenization challenges for low-resource, morphologically complex languages, but it is incremental as it builds on existing subword methods with supervised morphological analysis.
The authors tackled the problem of subword tokenization failing to preserve morphological boundaries in low-resource Geez script languages by developing MoVoC, a morphology-aware tokenizer, which showed consistent improvements in intrinsic metrics like MorphoScore and Boundary Precision, though not in translation quality.
Subword-based tokenization methods often fail to preserve morphological boundaries, a limitation especially pronounced in low-resource, morphologically complex languages such as those written in the Geez script. To address this, we present MoVoC (Morpheme-aware Subword Vocabulary Construction) and train MoVoC-Tok, a tokenizer that integrates supervised morphological analysis into the subword vocabulary. This hybrid segmentation approach combines morpheme-based and Byte Pair Encoding (BPE) tokens to preserve morphological integrity while maintaining lexical meaning. To tackle resource scarcity, we curate and release manually annotated morpheme data for four Geez script languages and a morpheme-aware vocabulary for two of them. While the proposed tokenization method does not lead to significant gains in automatic translation quality, we observe consistent improvements in intrinsic metrics, MorphoScore, and Boundary Precision, highlighting the value of morphology-aware segmentation in enhancing linguistic fidelity and token efficiency. Our morpheme-annotated datasets and tokenizer will be publicly available to support further research in low-resource, morphologically rich languages. Our code and data are available on GitHub: https://github.com/hailaykidu/MoVoC