Beyond Human-Level Accuracy: Computational Challenges in Deep Learning
This addresses computational bottlenecks for scaling deep learning to frontier accuracy targets, with implications for hardware design, though it is incremental as it builds on prior work.
The paper projects the dataset and model size growth needed to achieve beyond human-level accuracy in deep learning, estimating datasets must grow 33–971 times and models 6.6–456 times, and identifies distinct computational challenges for RNNs compared to CNNs, such as large memory footprints.
Deep learning (DL) research yields accuracy and product improvements from both model architecture changes and scale: larger data sets and models, and more computation. For hardware design, it is difficult to predict DL model changes. However, recent prior work shows that as dataset sizes grow, DL model accuracy and model size grow predictably. This paper leverages the prior work to project the dataset and model size growth required to advance DL accuracy beyond human-level, to frontier targets defined by machine learning experts. Datasets will need to grow $33$--$971 \times$, while models will need to grow $6.6$--$456\times$ to achieve target accuracies. We further characterize and project the computational requirements to train these applications at scale. Our characterization reveals an important segmentation of DL training challenges for recurrent neural networks (RNNs) that contrasts with prior studies of deep convolutional networks. RNNs will have comparatively moderate operational intensities and very large memory footprint requirements. In contrast to emerging accelerator designs, large-scale RNN training characteristics suggest designs with significantly larger memory capacity and on-chip caches.