Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languages
This work addresses the problem of understanding LM inductive biases for linguists and AI researchers, but it is incremental as it extends prior studies with a more formal grammar and a focus on length generalization.
The paper investigates whether language models (LMs) have inductive biases favoring typologically frequent grammatical properties, using Generalized Categorial Grammar (GCG) to create artificial languages that better capture natural language features. The result shows that typologically plausible word orders tend to facilitate LMs' generalization to longer, unseen sentences.
Whether language models (LMs) have inductive biases that favor typologically frequent grammatical properties over rare, implausible ones has been investigated, typically using artificial languages (ALs) (White and Cotterell, 2021; Kuribayashi et al., 2024). In this paper, we extend these works from two perspectives. First, we extend their context-free AL formalization by adopting Generalized Categorial Grammar (GCG) (Wood, 2014), which allows ALs to cover attested but previously overlooked constructions, such as unbounded dependency and mildly context-sensitive structures. Second, our evaluation focuses more on the generalization ability of LMs to process unseen longer test sentences. Thus, our ALs better capture features of natural languages and our experimental paradigm leads to clearer conclusions -- typologically plausible word orders tend to be easier for LMs to productively generalize.