Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics
This work addresses the problem of data quality evaluation for NLP researchers, offering a diagnostic tool that is incremental in leveraging existing training information.
The authors tackled the challenge of assessing data quality in large NLP datasets by introducing Data Maps, a tool that uses training dynamics to categorize instances into ambiguous, easy-to-learn, and hard-to-learn regions, revealing that ambiguous instances boost out-of-distribution generalization and hard-to-learn ones often indicate labeling errors.
Large datasets have become commonplace in NLP research. However, the increased emphasis on data quantity has made it challenging to assess the quality of data. We introduce Data Maps---a model-based tool to characterize and diagnose datasets. We leverage a largely ignored source of information: the behavior of the model on individual instances during training (training dynamics) for building data maps. This yields two intuitive measures for each example---the model's confidence in the true class, and the variability of this confidence across epochs---obtained in a single run of training. Experiments across four datasets show that these model-dependent measures reveal three distinct regions in the data map, each with pronounced characteristics. First, our data maps show the presence of "ambiguous" regions with respect to the model, which contribute the most towards out-of-distribution generalization. Second, the most populous regions in the data are "easy to learn" for the model, and play an important role in model optimization. Finally, data maps uncover a region with instances that the model finds "hard to learn"; these often correspond to labeling errors. Our results indicate that a shift in focus from quantity to quality of data could lead to robust models and improved out-of-distribution generalization.