What is the Best Sequence Length for BABYLM?
This work addresses sequence length optimization for small-scale language models, providing incremental insights for researchers in low-resource NLP.
The study investigated the impact of sequence length on pretraining for the BabyLM Challenge, finding that optimal lengths vary by task and architecture, with shorter sequences sufficing for grammatical generalization and longer contexts aiding morphological analogical reasoning.
Transformer language models typically operate with a fixed-length context window, which has grown in step with large-scale pretraining datasets. In the BabyLM Challenge, however, many past submissions have defaulted to using much shorter sequence lengths. We examine the impact of sequence length on BabyLM pretraining, to answer the simple question: what sequence length should we be using when training Baby LMs? Using 100M-word training data and fixed compute budgets, we compare 125M-parameter Mamba and OPT models, finding that although longer is often better, the optimal length depends on both task and architecture. Shorter sequences are sufficient for grammatical generalization tasks whereas longer contexts benefit morphological analogical reasoning tasks.