Mean BERTs make erratic language teachers: the effectiveness of latent bootstrapping in low-resource settings
This addresses the challenge of training language models with limited data, which is incremental as it builds on existing self-supervision methods.
The paper tackles the problem of pretraining language models in low-resource settings by exploring latent bootstrapping as an alternative self-supervision technique, and finds it effective for acquiring linguistic knowledge, with experiments showing competitive results on four linguistic benchmarks using small curated corpora.
This paper explores the use of latent bootstrapping, an alternative self-supervision technique, for pretraining language models. Unlike the typical practice of using self-supervision on discrete subwords, latent bootstrapping leverages contextualized embeddings for a richer supervision signal. We conduct experiments to assess how effective this approach is for acquiring linguistic knowledge from limited resources. Specifically, our experiments are based on the BabyLM shared task, which includes pretraining on two small curated corpora and an evaluation on four linguistic benchmarks.