Efficacy of Synthetic Data as a Benchmark
This addresses the reliability of synthetic benchmarks for NLP practitioners, highlighting task-dependent limitations and biases, but it is incremental as it builds on existing synthetic data research.
The study investigated the effectiveness of using synthetic data generated by large language models as benchmarks for NLP tasks, finding that it works well for simpler tasks like intent classification but falls short for complex ones like named entity recognition, with smaller models showing biases toward their own data.
Large language models (LLMs) have enabled a range of applications in zero-shot and few-shot learning settings, including the generation of synthetic datasets for training and testing. However, to reliably use these synthetic datasets, it is essential to understand how representative they are of real-world data. We investigate this by assessing the effectiveness of generating synthetic data through LLM and using it as a benchmark for various NLP tasks. Our experiments across six datasets, and three different tasks, show that while synthetic data can effectively capture performance of various methods for simpler tasks, such as intent classification, it falls short for more complex tasks like named entity recognition. Additionally, we propose a new metric called the bias factor, which evaluates the biases introduced when the same LLM is used to both generate benchmarking data and to perform the tasks. We find that smaller LLMs exhibit biases towards their own generated data, whereas larger models do not. Overall, our findings suggest that the effectiveness of synthetic data as a benchmark varies depending on the task, and that practitioners should rely on data generated from multiple larger models whenever possible.