Morphology Matters: A Multilingual Language Modeling Analysis
This study resolves a disagreement in prior research regarding the effect of inflectional morphology on language modeling for NLP researchers working with multilingual models, offering insights into better segmentation strategies.
The paper investigates the impact of inflectional morphology on multilingual language modeling performance using a corpus of 145 Bible translations in 92 languages. It finds that several morphological measures are significantly associated with higher surprisal in LSTM models trained with BPE-segmented data. The study also shows that linguistically-motivated subword segmentation strategies like Morfessor and FSTs improve performance and mitigate the negative impact of morphology.
Prior studies in multilingual language modeling (e.g., Cotterell et al., 2018; Mielke et al., 2019) disagree on whether or not inflectional morphology makes languages harder to model. We attempt to resolve the disagreement and extend those studies. We compile a larger corpus of 145 Bible translations in 92 languages and a larger number of typological features. We fill in missing typological data for several languages and consider corpus-based measures of morphological complexity in addition to expert-produced typological features. We find that several morphological measures are significantly associated with higher surprisal when LSTM models are trained with BPE-segmented data. We also investigate linguistically-motivated subword segmentation strategies like Morfessor and Finite-State Transducers (FSTs) and find that these segmentation strategies yield better performance and reduce the impact of a language's morphology on language modeling.