Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuning
This work addresses the efficiency and cost issues in data filtering for instruction tuning, offering a practical solution for researchers and practitioners, though it is incremental as it builds on existing filtering methods.
The paper tackles the problem of high computational cost in data filtering for instruction tuning of large language models by proposing Superfiltering, which uses a smaller, weaker model to filter data for a larger model, achieving faster filtering and improved performance on standard benchmarks.
Instruction tuning is critical to improve LLMs but usually suffers from low-quality and redundant data. Data filtering for instruction tuning has proved important in improving both the efficiency and performance of the tuning process. But it also leads to extra cost and computation due to the involvement of LLMs in this process. To reduce the filtering cost, we study Superfiltering: Can we use a smaller and weaker model to select data for finetuning a larger and stronger model? Despite the performance gap between weak and strong language models, we find their highly consistent capability to perceive instruction difficulty and data selection results. This enables us to use a much smaller and more efficient model to filter the instruction data used to train a larger language model. Not only does it largely speed up the data filtering, but the filtered-data-finetuned LLM achieves even better performance on standard benchmarks. Extensive experiments validate the efficacy and efficiency of our approach.