Specialist or Generalist? Instruction Tuning for Specific NLP Tasks
This work addresses the problem of optimizing instruction tuning for building specialist NLP models, providing a systematic guide for researchers and practitioners, though it is incremental as it builds on existing methods.
The paper investigates whether incorporating broad-coverage generalist instruction tuning can enhance specialist models for specific NLP tasks, finding that it consistently improves performance when task coverage is broad, especially with limited task-specific data, but may harm tasks requiring factual knowledge due to hallucinatory information.
The potential of large language models (LLMs) to simultaneously perform a wide range of natural language processing (NLP) tasks has been the subject of extensive research. Although instruction tuning has proven to be a data-efficient method for transforming LLMs into such generalist models, their performance still lags behind specialist models trained exclusively for specific tasks. In this paper, we investigate whether incorporating broad-coverage generalist instruction tuning can contribute to building a specialist model. We hypothesize that its efficacy depends on task specificity and skill requirements. Our experiments assess four target tasks with distinct coverage levels, revealing that integrating generalist instruction tuning consistently enhances model performance when the task coverage is broad. The effect is particularly pronounced when the amount of task-specific training data is limited. Further investigation into three target tasks focusing on different capabilities demonstrates that generalist instruction tuning improves understanding and reasoning abilities. However, for tasks requiring factual knowledge, generalist data containing hallucinatory information may negatively affect the model's performance. Overall, our work provides a systematic guide for developing specialist models with general instruction tuning. Our code and other related resources can be found at https://github.com/DavidFanzz/Generalist_or_Specialist.