FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
This addresses the data scarcity problem for instruction-tuning in LLMs, offering a scalable solution to improve model alignment with user prompts, though it is incremental in leveraging existing data sources.
The authors tackled the limited supervised training data for large language models by proposing FineInstructions, a method to generate billions of synthetic instruction-answer pairs from internet-scale pre-training documents, enabling pre-training solely with an instruction-tuning objective. They found that pre-training on FineInstructions outperforms standard pre-training and other synthetic techniques on benchmarks measuring free-form response quality.
Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of "instruction-tuning" data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts. These instruction templates are matched to and instantiated with human-written source documents from unstructured pre-training corpora. With "supervised" synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at https://huggingface.co/fineinstructions .