Disentangling the Roles of Representation and Selection in Data Pruning
This work addresses the efficiency of NLP model training by systematically analyzing data pruning methods, though it is incremental as it focuses on understanding existing design choices rather than introducing new techniques.
The paper tackles the problem of data pruning in NLP by decomposing it into representation and selection components, finding that better representations like training gradients improve instance selection regardless of algorithm, and that selection algorithms vary in performance and do not always meet their objectives.
Data pruning, selecting small but impactful subsets, offers a promising way to efficiently scale NLP model training. However, existing methods often involve many different design choices, which have not been systematically studied. This limits future developments. In this work, we decompose data pruning into two key components: the data representation and the selection algorithm, and we systematically analyze their influence on the selection of instances. Our theoretical and empirical results highlight the crucial role of representations: better representations, e.g., training gradients, generally lead to a better selection of instances, regardless of the chosen selection algorithm. Furthermore, different selection algorithms excel in different settings, and none consistently outperforms the others. Moreover, the selection algorithms do not always align with their intended objectives: for example, algorithms designed for the same objective can select drastically different instances, highlighting the need for careful evaluation.