Technical Report: Competition Solution For BetterMixture
This work addresses the challenge of efficient data mixing for fine-tuning large language models, but it is incremental as it builds on existing tools like Data-Juicer.
The paper tackled the problem of selecting and optimizing datasets to enhance large language model performance under limited computational resources, achieving third place in the BetterMixture challenge by using data deduplication, quality filtering, and diversity selection.
In the era of flourishing large-scale models, the challenge of selecting and optimizing datasets from the vast and complex sea of data, to enhance the performance of large language models within the constraints of limited computational resources, has become paramount. This paper details our solution for the BetterMixture challenge, which focuses on the fine-tuning data mixing for large language models. Our approach, which secured third place, incorporates data deduplication, low-level and high-level quality filtering, and diversity selection. The foundation of our solution is Ke-Data-Juicer, an extension of Data-Juicer, demonstrating its robust capabilities in handling and optimizing data for large language models.