ChapGTP, ILLC's Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formation
This work addresses data efficiency for low-resource language modeling, presenting an incremental improvement with a novel data augmentation technique.
The researchers tackled the problem of training language models with limited data by developing ChapGTP, a masked language model enhanced with Automatic Task Formation, which achieved competitive results on evaluation suites like BLiMP, (Super)GLUE, and MSGS in the BabyLM challenge.
We present the submission of the ILLC at the University of Amsterdam to the BabyLM challenge (Warstadt et al., 2023), in the strict-small track. Our final model, ChapGTP, is a masked language model that was trained for 200 epochs, aided by a novel data augmentation technique called Automatic Task Formation. We discuss in detail the performance of this model on the three evaluation suites: BLiMP, (Super)GLUE, and MSGS. Furthermore, we present a wide range of methods that were ultimately not included in the model, but may serve as inspiration for training LMs in low-resource settings.