A State-Vector Framework for Dataset Effects
This work addresses the underexplored issue of dataset interactions for researchers and practitioners in machine learning, though it is incremental as it builds on existing probing methods.
The paper tackles the problem of understanding how datasets interact and affect deep neural network training by proposing a state-vector framework to quantify these effects, showing that some language datasets have characteristic impacts concentrated on specific linguistic dimensions and exhibit spill-over effects.
The impressive success of recent deep neural network (DNN)-based systems is significantly influenced by the high-quality datasets used in training. However, the effects of the datasets, especially how they interact with each other, remain underexplored. We propose a state-vector framework to enable rigorous studies in this direction. This framework uses idealized probing test results as the bases of a vector space. This framework allows us to quantify the effects of both standalone and interacting datasets. We show that the significant effects of some commonly-used language understanding datasets are characteristic and are concentrated on a few linguistic dimensions. Additionally, we observe some ``spill-over'' effects: the datasets could impact the models along dimensions that may seem unrelated to the intended tasks. Our state-vector framework paves the way for a systematic understanding of the dataset effects, a crucial component in responsible and robust model development.