Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction
This addresses the problem of data scarcity for instruction-following in LLMs, offering a scalable and efficient solution for researchers and practitioners, though it is an incremental improvement over existing synthetic methods.
The paper tackles the challenge of synthesizing high-quality instruction-tuning data for LLMs by proposing Web Reconstruction (WebR), a fully automated framework that generates data from raw web documents without relying on seed data or strong assumptions, resulting in datasets that outperform state-of-the-art baselines by up to 16.65% across four benchmarks.
The improvement of LLMs' instruction-following capabilities depends critically on the availability of high-quality instruction-response pairs. While existing automatic data synthetic methods alleviate the burden of manual curation, they often rely heavily on either the quality of seed data or strong assumptions about the structure and content of web documents. To tackle these challenges, we propose Web Reconstruction (WebR), a fully automated framework for synthesizing high-quality instruction-tuning (IT) data directly from raw web documents with minimal assumptions. Leveraging the inherent diversity of raw web content, we conceptualize web reconstruction as an instruction-tuning data synthesis task via a novel dual-perspective paradigm--Web as Instruction and Web as Response--where each web document is designated as either an instruction or a response to trigger the reconstruction process. Comprehensive experiments show that datasets generated by WebR outperform state-of-the-art baselines by up to 16.65% across four instruction-following benchmarks. Notably, WebR demonstrates superior compatibility, data efficiency, and scalability, enabling enhanced domain adaptation with minimal effort. The data and code are publicly available at https://github.com/YJiangcm/WebR.