The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages
This addresses the challenge of improving text generation and cross-lingual transfer for low-resource, morphologically complex languages, though it is incremental as it extends an existing framework.
The paper studied how subword segmentation evolves during pretraining and finetuning when learned dynamically by language models, finding that morphologically complex languages like Isi-Xhosa show greater instability and subword boundaries become finer-grained in finetuning.
Subword segmentation is typically applied in preprocessing and stays fixed during training. Alternatively, it can be learned during training to optimise the training objective. In this paper we study the learning dynamics of subword segmentation: if a language model can dynamically optimise tokenisation, how do its subwords evolve during pretraining and finetuning? To explore this, we extend the subword segmental language model (SSLM), a framework for learning subwords during training, to support pretraining and finetuning. We train models for three typologically diverse languages to study learning dynamics across the morphological spectrum: Isi-Xhosa is conjunctive (long word forms composed of many morphemes), Setswana is disjunctive (morphemes written as separate words), and English represents a typological middle ground. We analyse subword dynamics from a linguistic perspective, tracking morphology, productivity, and fertility. We identify four stages of subword learning, with the morphologically complex isi-Xhosa exhibiting greater instability. During finetuning, subword boundaries shift to become finer-grained. Lastly, we show that learnable subwords offers a promising approach to improve text generation and cross-lingual transfer for low-resource, morphologically complex languages.