SearchInstruct: Enhancing Domain Adaptation via Retrieval-Based Instruction Dataset Creation
This addresses the problem of data scarcity and domain constraints for researchers and practitioners adapting LLMs to specialized fields, representing an incremental advancement in dataset creation methods.
The paper tackles the challenge of creating domain-specific instruction datasets for supervised fine-tuning of large language models by proposing SearchInstruct, a retrieval-based method that expands human-generated questions and retrieves domain resources to generate answers, resulting in measurable improvements in LLM performance in specialized domains.
Supervised Fine-Tuning (SFT) is essential for training large language models (LLMs), significantly enhancing critical capabilities such as instruction following and in-context learning. Nevertheless, creating suitable training datasets tailored for specific domains remains challenging due to unique domain constraints and data scarcity. In this paper, we propose SearchInstruct, an innovative method explicitly designed to construct high quality instruction datasets for SFT. Our approach begins with a limited set of domain specific, human generated questions, which are systematically expanded using a large language model. Subsequently, domain relevant resources are dynamically retrieved to generate accurate and contextually appropriate answers for each augmented question. Experimental evaluation demonstrates that SearchInstruct enhances both the diversity and quality of SFT datasets, leading to measurable improvements in LLM performance within specialized domains. Additionally, we show that beyond dataset generation, the proposed method can also effectively facilitate tasks such as model editing, enabling efficient updates to existing models. To facilitate reproducibility and community adoption, we provide full implementation details, the complete set of generated instruction response pairs, and the source code in a publicly accessible Git repository: [https://github.com/mostafaamiri/SearchInstruct](https://github.com/mostafaamiri/SearchInstruct)