Stronger Models are NOT Stronger Teachers for Instruction Tuning
This work addresses a key problem in synthetic instruction dataset generation for LLM instruction tuning, revealing an incremental insight that could optimize resource allocation and model training efficiency.
The paper challenges the assumption that larger models are better teachers for instruction tuning, showing through experiments across five base models and twenty response generators that stronger models do not necessarily improve smaller models' performance, a phenomenon termed the Larger Models' Paradox.
Instruction tuning has been widely adopted to ensure large language models (LLMs) follow user instructions effectively. The resulting instruction-following capabilities of LLMs heavily rely on the instruction datasets used for tuning. Recently, synthetic instruction datasets have emerged as an economically viable solution to provide LLMs diverse and high-quality instructions. However, existing approaches typically assume that larger or stronger models are stronger teachers for instruction tuning, and hence simply adopt these models as response generators to the synthetic instructions. In this paper, we challenge this commonly-adopted assumption. Our extensive experiments across five base models and twenty response generators reveal that larger and stronger models are not necessarily stronger teachers of smaller models. We refer to this phenomenon as the Larger Models' Paradox. We observe that existing metrics cannot precisely predict the effectiveness of response generators since they ignore the compatibility between teachers and base models being fine-tuned. We thus develop a novel metric, named as Compatibility-Adjusted Reward (CAR) to measure the effectiveness of response generators. Our experiments across five base models demonstrate that CAR outperforms almost all baselines.