DUET: Optimizing Training Data Mixtures via Feedback from Unseen Evaluation Tasks
This addresses a practical challenge for LLM developers in fine-tuning models without access to task-specific data, though it is incremental as it builds on prior data selection and mixing techniques.
The paper tackles the problem of optimizing training data mixtures for LLMs when evaluation task data is unknown, by proposing DUET, a global-to-local algorithm that uses feedback from unseen tasks to select data, and demonstrates improved performance over existing methods in experiments across various language tasks.
The performance of an LLM depends heavily on the relevance of its training data to the downstream evaluation task. However, in practice, the data involved in an unseen evaluation task is often unknown (e.g., conversations between an LLM and a user are end-to-end encrypted). Hence, it is unclear what data are relevant for fine-tuning the LLM to maximize its performance on the specific unseen evaluation task. Instead, one can only deploy the LLM on the unseen task to gather multiple rounds of feedback on how well the model performs (e.g., user ratings). This novel setting offers a refreshing perspective towards optimizing training data mixtures via feedback from an unseen evaluation task, which prior data mixing and selection works do not consider. Our paper presents DUET, a novel global-to-local algorithm that interleaves influence function as a data selection method with Bayesian optimization to optimize data mixture via feedback from a specific unseen evaluation task. By analyzing DUET's cumulative regret, we theoretically show that DUET converges to the optimal training data mixture for an unseen task even without any data knowledge of the task. Finally, our experiments across a variety of language tasks demonstrate that DUET outperforms existing data selection and mixing methods in the unseen-task setting.