MAmmoTH2: Scaling Instructions from the Web
This provides a new paradigm for generating high-quality instruction data without costly human or GPT-4 annotation, benefiting researchers and developers in AI and NLP by enabling more scalable and efficient model training.
The authors tackled the challenge of scaling instruction tuning for large language models by efficiently harvesting 10 million instruction-response pairs from web corpora, leading to MAmmoTH2 models that significantly improved reasoning performance, such as boosting a 7B model from 11% to 36.7% on MATH and from 36% to 68.4% on GSM8K.
Instruction tuning improves the reasoning abilities of large language models (LLMs), with data quality and scalability being the crucial factors. Most instruction tuning data come from human crowd-sourcing or GPT-4 distillation. We propose a paradigm to efficiently harvest 10 million naturally existing instruction data from the pre-training web corpus to enhance LLM reasoning. Our approach involves (1) recalling relevant documents, (2) extracting instruction-response pairs, and (3) refining the extracted pairs using open-source LLMs. Fine-tuning base LLMs on this dataset, we build MAmmoTH2 models, which significantly boost performance on reasoning benchmarks. Notably, MAmmoTH2-7B's (Mistral) performance increases from 11% to 36.7% on MATH and from 36% to 68.4% on GSM8K without training on any in-domain data. Further training MAmmoTH2 on public instruction tuning datasets yields MAmmoTH2-Plus, achieving state-of-the-art performance on several reasoning and chatbot benchmarks. Our work demonstrates how to harvest large-scale, high-quality instruction data without costly human annotation or GPT-4 distillation, providing a new paradigm for building better instruction tuning data.