Synthetic bootstrapped pretraining
This addresses the challenge of enhancing language model performance through better inter-document correlation modeling, offering a novel pretraining approach with strong empirical gains, though it is incremental in the context of existing pretraining methods.
The paper tackles the problem of inefficient modeling of inter-document correlations in language model pretraining by introducing Synthetic Bootstrapped Pretraining (SBP), which learns relations between documents to synthesize a vast new corpus for joint training, resulting in a 3B-parameter model pretrained on up to 1T tokens that consistently improves upon a baseline and achieves a significant fraction of performance improvement attainable by an oracle with 20x more unique data.
We introduce Synthetic Bootstrapped Pretraining (SBP), a language model (LM) pretraining procedure that first learns a model of relations between documents from the pretraining dataset and then leverages it to synthesize a vast new corpus for joint training. While the standard pretraining teaches LMs to learn causal correlations among tokens within a single document, it is not designed to efficiently model the rich, learnable inter-document correlations that can potentially lead to better performance. We validate SBP by designing a compute-matched pretraining setup and pretrain a 3B-parameter model on up to 1T tokens from scratch. We find SBP consistently improves upon a strong repetition baseline and delivers a significant fraction of performance improvement attainable by an oracle upper bound with access to 20x more unique data. Qualitative analysis reveals that the synthesized documents go beyond mere paraphrases -- SBP first abstracts a core concept from the seed material and then crafts a new narration on top of it. Besides strong empirical performance, SBP admits a natural Bayesian interpretation: the synthesizer implicitly learns to abstract the latent concepts shared between related documents.