Optimizing Data Collection for Machine Learning
This addresses the challenge of inefficient data collection for ML practitioners, though it is incremental in applying optimization to a known bottleneck.
The paper tackles the problem of determining optimal data collection for machine learning by modeling it as a formal optimization problem, and it shows that their method significantly reduces the risk of failing to meet performance targets while keeping costs low on tasks like classification and segmentation.
Modern deep learning systems require huge data sets to achieve impressive performance, but there is little guidance on how much or what kind of data to collect. Over-collecting data incurs unnecessary present costs, while under-collecting may incur future costs and delay workflows. We propose a new paradigm for modeling the data collection workflow as a formal optimal data collection problem that allows designers to specify performance targets, collection costs, a time horizon, and penalties for failing to meet the targets. Additionally, this formulation generalizes to tasks requiring multiple data sources, such as labeled and unlabeled data used in semi-supervised learning. To solve our problem, we develop Learn-Optimize-Collect (LOC), which minimizes expected future collection costs. Finally, we numerically compare our framework to the conventional baseline of estimating data requirements by extrapolating from neural scaling laws. We significantly reduce the risks of failing to meet desired performance targets on several classification, segmentation, and detection tasks, while maintaining low total collection costs.