LLM Data Selection and Utilization via Dynamic Bi-level Optimization
This work addresses the challenge of reducing computational costs and enhancing training efficiency for LLM developers, representing an incremental improvement over static data selection methods.
The paper tackles the problem of inefficient data selection in large language model training by introducing a dynamic bi-level optimization framework that adjusts data weights during training, resulting in improved model performance and transferable weighting models.
While large-scale training data is fundamental for developing capable large language models (LLMs), strategically selecting high-quality data has emerged as a critical approach to enhance training efficiency and reduce computational costs. Current data selection methodologies predominantly rely on static, training-agnostic criteria, failing to account for the dynamic model training and data interactions. In this paper, we propose a new Data Weighting Model (DWM) to adjust the weight of selected data within each batch to achieve a dynamic data utilization during LLM training. Specially, to better capture the dynamic data preference of the trained model, a bi-level optimization framework is implemented to update the weighting model. Our experiments demonstrate that DWM enhances the performance of models trained with randomly-selected data, and the learned weighting model can be transferred to enhance other data selection methods and models of different sizes. Moreover, we further analyze how a model's data preferences evolve throughout training, providing new insights into the data preference of the model during training.