Instruction Matters: A Simple yet Effective Task Selection for Optimized Instruction Tuning of Specific Tasks
This addresses the efficiency problem in instruction tuning for NLP practitioners by providing a simpler alternative to complex transferability measurements or data sample creation.
The paper tackles the problem of selecting relevant tasks for instruction tuning to improve performance on specific tasks, introducing a method that uses only instruction information to identify relevant tasks. Experimental results show substantial performance improvements on multiple benchmarks (P3, Big-Bench, NIV2, Big-Bench Hard) that surpass prior task selection methods.
Instruction tuning has been proven effective in enhancing zero-shot generalization across various tasks and in improving the performance of specific tasks. For task-specific improvements, strategically selecting and training on related tasks that provide meaningful supervision is crucial, as this approach enhances efficiency and prevents performance degradation from learning irrelevant tasks. In this light, we introduce a simple yet effective task selection method that leverages instruction information alone to identify relevant tasks, optimizing instruction tuning for specific tasks. Our method is significantly more efficient than traditional approaches, which require complex measurements of pairwise transferability between tasks or the creation of data samples for the target task. Additionally, by aligning the model with the unique instructional template style of the meta-dataset, we enhance its ability to granularly discern relevant tasks, leading to improved overall performance. Experimental results demonstrate that training on a small set of tasks, chosen solely based on the instructions, results in substantial improvements in performance on benchmarks such as P3, Big-Bench, NIV2, and Big-Bench Hard. Significantly, these improvements surpass those achieved by prior task selection methods, highlighting the superiority of our approach.