How Much More Data Do I Need? Estimating Requirements for Downstream Tasks
This work addresses a critical issue for practitioners in fields like autonomous driving and medical imaging, where data collection is costly, by providing incremental improvements to data requirement estimation methods.
The paper tackles the problem of estimating how much additional training data is needed to achieve a target performance in downstream tasks, finding that existing power-law scaling laws are insufficient and proposing a generalized function family with a tuned correction factor and multi-round collection to improve accuracy, leading to significant savings in development time and data acquisition costs.
Given a small training data set and a learning algorithm, how much more data is necessary to reach a target validation or test performance? This question is of critical importance in applications such as autonomous driving or medical imaging where collecting data is expensive and time-consuming. Overestimating or underestimating data requirements incurs substantial costs that could be avoided with an adequate budget. Prior work on neural scaling laws suggest that the power-law function can fit the validation performance curve and extrapolate it to larger data set sizes. We find that this does not immediately translate to the more difficult downstream task of estimating the required data set size to meet a target performance. In this work, we consider a broad class of computer vision tasks and systematically investigate a family of functions that generalize the power-law function to allow for better estimation of data requirements. Finally, we show that incorporating a tuned correction factor and collecting over multiple rounds significantly improves the performance of the data estimators. Using our guidelines, practitioners can accurately estimate data requirements of machine learning systems to gain savings in both development time and data acquisition costs.