A Proposal to Study "Is High Quality Data All We Need?"
This addresses the challenge of data efficiency and robustness in AI for researchers and practitioners, but it is incremental as it builds on existing data-centric approaches.
The paper tackles the problem of deep neural networks failing to generalize to out-of-distribution or adversarial datasets by hypothesizing that smaller, high-quality datasets might replace large-scale ones, proposing an empirical study to investigate data pruning and creation methods for generating such datasets.
Even though deep neural models have achieved superhuman performance on many popular benchmarks, they have failed to generalize to OOD or adversarial datasets. Conventional approaches aimed at increasing robustness include developing increasingly large models and augmentation with large scale datasets. However, orthogonal to these trends, we hypothesize that a smaller, high quality dataset is what we need. Our hypothesis is based on the fact that deep neural networks are data driven models, and data is what leads/misleads models. In this work, we propose an empirical study that examines how to select a subset of and/or create high quality benchmark data, for a model to learn effectively. We seek to answer if big datasets are truly needed to learn a task, and whether a smaller subset of high quality data can replace big datasets. We plan to investigate both data pruning and data creation paradigms to generate high quality datasets.