On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibility
This work addresses the potential architectural bias in language models for non-English languages, but results are incremental as they challenge prior findings without offering new solutions.
The study investigated whether positional encodings in language models affect performance differently across languages based on morphological complexity and word order flexibility, finding no clear interaction and emphasizing the importance of task, language, and metric choices.
Language model architectures are predominantly first created for English and subsequently applied to other languages. It is an open question whether this architectural bias leads to degraded performance for languages that are structurally different from English. We examine one specific architectural choice: positional encodings, through the lens of the trade-off hypothesis: the supposed interplay between morphological complexity and word order flexibility. This hypothesis posits a trade-off between the two: a more morphologically complex language can have a more flexible word order, and vice-versa. Positional encodings are a direct target to investigate the implications of this hypothesis in relation to language modelling. We pretrain monolingual model variants with absolute, relative, and no positional encodings for seven typologically diverse languages and evaluate them on four downstream tasks. Contrary to previous findings, we do not observe a clear interaction between position encodings and morphological complexity or word order flexibility, as measured by various proxies. Our results show that the choice of tasks, languages, and metrics are essential for drawing stable conclusions