A surprisal oracle for when every layer counts
This is an incremental improvement to a learner-directed training approach for language models, specifically targeting the BabyLM competition.
The authors proposed an updated Active Curriculum Language Modeling (ACLM) process for the BabyLM 2024 task, which uses a dynamic similarity model to prioritize uncertain training items. They found that while their models underperformed on grammatical inferences, they outperformed official baselines on common-sense and world-knowledge tasks.
Active Curriculum Language Modeling (ACLM; Hong et al., 2023) is a learner directed approach to training a language model. We proposed the original version of this process in our submission to the BabyLM 2023 task, and now we propose an updated ACLM process for the BabyLM 2024 task. ACLM involves an iteratively- and dynamically-constructed curriculum informed over the training process by a model of uncertainty; other training items that are similarly uncertain to a least certain candidate item are prioritized. Our new process improves the similarity model so that it is more dynamic, and we run ACLM over the most successful model from the BabyLM 2023 task: ELC-BERT (Charpentier and Samuel, 2023). We find that while our models underperform on fine-grained grammatical inferences, they outperform the BabyLM 2024 official base-lines on common-sense and world-knowledge tasks. We make our code available at https: //github.com/asayeed/ActiveBaby.