ToddlerBERTa: Exploiting BabyBERTa for Grammar Learning and Language Understanding
This work addresses the problem of efficient language model training for researchers and practitioners, but it is incremental as it builds on existing BabyBERTa-like models.
The researchers investigated whether smaller language models like ToddlerBERTa, trained on limited data, could achieve strong performance in grammar and language understanding tasks, finding that they rival state-of-the-art models like RoBERTa-base on benchmarks such as BLiMP and SuperGLUE.
We present ToddlerBERTa, a BabyBERTa-like language model, exploring its capabilities through five different models with varied hyperparameters. Evaluating on BLiMP, SuperGLUE, MSGS, and a Supplement benchmark from the BabyLM challenge, we find that smaller models can excel in specific tasks, while larger models perform well with substantial data. Despite training on a smaller dataset, ToddlerBERTa demonstrates commendable performance, rivalling the state-of-the-art RoBERTa-base. The model showcases robust language understanding, even with single-sentence pretraining, and competes with baselines that leverage broader contextual information. Our work provides insights into hyperparameter choices, and data utilization, contributing to the advancement of language models.