Algorithmic progress in computer vision
This provides insights into the drivers of progress in computer vision for researchers and practitioners, though it is incremental in analyzing existing trends.
The paper investigates algorithmic progress in computer vision, specifically on ImageNet, and finds that algorithmic improvements have been as important as compute scaling, with compute-augmenting innovations halving compute requirements every nine months.
We investigate algorithmic progress in image classification on ImageNet, perhaps the most well-known test bed for computer vision. We estimate a model, informed by work on neural scaling laws, and infer a decomposition of progress into the scaling of compute, data, and algorithms. Using Shapley values to attribute performance improvements, we find that algorithmic improvements have been roughly as important as the scaling of compute for progress computer vision. Our estimates indicate that algorithmic innovations mostly take the form of compute-augmenting algorithmic advances (which enable researchers to get better performance from less compute), not data-augmenting algorithmic advances. We find that compute-augmenting algorithmic advances are made at a pace more than twice as fast as the rate usually associated with Moore's law. In particular, we estimate that compute-augmenting innovations halve compute requirements every nine months (95\% confidence interval: 4 to 25 months).