A Little Human Data Goes A Long Way
This addresses the cost-effectiveness of human vs. synthetic data for NLP practitioners, showing incremental benefits of minimal human annotation in data-scarce scenarios.
The study investigated the impact of replacing human-annotated data with synthetic data in Fact Verification and Question Answering, finding that performance drops severely only when replacing the final 10% of human data, and that adding just 125 human data points significantly improves models trained on purely synthetic data.
Faced with an expensive human annotation process, creators of NLP systems increasingly turn to synthetic data generation. While this method shows promise, the extent to which synthetic data can replace human annotation is poorly understood. We investigate the use of synthetic data in Fact Verification (FV) and Question Answering (QA) by studying the effects of incrementally replacing human generated data with synthetic points on eight diverse datasets. Strikingly, replacing up to 90% of the training data only marginally decreases performance, but replacing the final 10% leads to severe declines. We find that models trained on purely synthetic data can be reliably improved by including as few as 125 human generated data points. We show that matching the performance gain of just a little additional human data (only 200 points) requires an order of magnitude more synthetic data and estimate price ratios at which human annotation would be a more cost-effective solution. Our results suggest that even when human annotation at scale is infeasible, there is great value to having a small proportion of the dataset being human generated.