Investigating the Impact of Data Selection Strategies on Language Model Performance
This work addresses data selection for language models, but it is incremental as it builds on existing methods without introducing a new paradigm.
The study investigated how different data selection strategies affect language model performance, finding that embedding-based neural features provide complementary benefits to n-gram features for aligning with target distributions.
Data selection is critical for enhancing the performance of language models, particularly when aligning training datasets with a desired target distribution. This study explores the effects of different data selection methods and feature types on model performance. We evaluate whether selecting data subsets can influence downstream tasks, whether n-gram features improve alignment with target distributions, and whether embedding-based neural features provide complementary benefits. Through comparative experiments using baseline random selection methods and distribution aligned approaches, we provide insights into the interplay between data selection strategies and model training efficacy. All code for this study can be found on \href{https://github.com/jgu13/HIR-Hybrid-Importance-Resampling-for-Language-Models}{github repository}.