Fine Tuning with Abnormal Examples
This addresses the problem of dataset inefficiency for NLP practitioners, though it appears incremental as it builds on existing fine-tuning methods.
The paper tackles the problem of inefficient fine-tuning on large NLP datasets by identifying and using a subset of examples that create a more uniform distribution, achieving better performance with 10,500 examples than with all 87,000 examples in the SQUAD dataset.
Given the prevalence of crowd sourced labor in creating Natural Language processing datasets, these aforementioned sets have become increasingly large. For instance, the SQUAD dataset currently sits at over 80,000 records. However, because the English language is rather repetitive in structure, the distribution of word frequencies in the SQUAD dataset's contexts are relatively unchanged. By measuring each sentences distance from the co-variate distance of frequencies of all sentences in the dataset, we identify 10,500 examples that create a more uniform distribution for training. While fine-tuning ELECTRA [4] on this subset of examples reaches better performance to a model trained on all 87,000 examples. Herein we introduce a methodology for systematically pruning datasets for fine tuning reaching better out of sample performance.