A Morphology-Based Investigation of Positional Encodings
This work addresses a fundamental question in NLP about how language structure affects model components, potentially guiding better positional encoding designs for diverse languages.
The study investigated whether morphological complexity in languages correlates with the importance of positional encodings in pre-trained language models, finding that positional encoding significance decreases as morphological complexity increases across 22 languages and 5 downstream tasks.
Contemporary deep learning models effectively handle languages with diverse morphology despite not being directly integrated into them. Morphology and word order are closely linked, with the latter incorporated into transformer-based models through positional encodings. This prompts a fundamental inquiry: Is there a correlation between the morphological complexity of a language and the utilization of positional encoding in pre-trained language models? In pursuit of an answer, we present the first study addressing this question, encompassing 22 languages and 5 downstream tasks. Our findings reveal that the importance of positional encoding diminishes with increasing morphological complexity in languages. Our study motivates the need for a deeper understanding of positional encoding, augmenting them to better reflect the different languages under consideration.