Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis
This work addresses a data bottleneck for researchers and practitioners training LLMs on NLU tasks, though it is incremental as it builds on existing instruction synthesis methods.
The paper tackles the shortage of diverse, high-quality instructions for natural language understanding (NLU) tasks in large language models (LLMs) by proposing Hum, a large-scale synthetic instruction corpus. The result shows that Hum improves NLU capabilities of six LLMs by an average of 3.1% across 5 NLU tasks and 28 general evaluation datasets, with no significant decline in other general capabilities.
High-quality, large-scale instructions are crucial for aligning large language models (LLMs), however, there is a severe shortage of instruction in the field of natural language understanding (NLU). Previous works on constructing NLU instructions mainly focus on information extraction (IE), neglecting tasks such as machine reading comprehension, question answering, and text classification. Furthermore, the lack of diversity in the data has led to a decreased generalization ability of trained LLMs in other NLU tasks and a noticeable decline in the fundamental model's general capabilities. To address this issue, we propose Hum, a large-scale, high-quality synthetic instruction corpus for NLU tasks, designed to enhance the NLU capabilities of LLMs. Specifically, Hum includes IE (either close IE or open IE), machine reading comprehension, text classification, and instruction generalist tasks, thereby enriching task diversity. Additionally, we introduce a human-LLMs collaborative mechanism to synthesize instructions, which enriches instruction diversity by incorporating guidelines, preference rules, and format variants. We conduct extensive experiments on 5 NLU tasks and 28 general capability evaluation datasets for LLMs. Experimental results show that Hum enhances the NLU capabilities of six LLMs by an average of 3.1\%, with no significant decline observed in other general capabilities.