A Unified Understanding of Offline Data Selection and Online Self-refining Generation for Post-training LLMs
This work addresses the challenge of efficiently fine-tuning LLMs for specific tasks, though it appears incremental as it builds on existing data selection and generation methods.
The paper tackles the problem of improving data quality for adapting large language models to downstream tasks by proposing a unified framework for offline data selection and online self-refining generation, demonstrating performance gains over baselines through experiments on quality enhancement and safety-aware fine-tuning.
Offline data selection and online self-refining generation, which enhance the data quality, are crucial steps in adapting large language models (LLMs) to specific downstream tasks. We tackle offline data selection and online self-refining generations through an optimization perspective. Specifically, bilevel data selection is used for offline data selection with respect to the validation dataset, and we treat online self-refining generation as a model adaptation step of selecting the model trained on current responses that best fits the validation data. Our framework offers a unified understanding of offline data selection and self-refining generation by assigning a learned data weight to each question and response, either explicitly or implicitly. For the first time, we theoretically demonstrate the effectiveness of the bilevel data selection framework and demonstrate its performance gains over unfiltered direct mixing baselines. By combining offline data with validation-weighted online generations, our method enhances fine-tuning performance. Experiments on quality enhancement and safety-aware LLM fine-tuning validate its effectiveness.