Accelerating Deep Learning Inference via Freezing
This addresses latency issues for real-world DNN inference applications, but it is incremental as it builds on existing caching and approximation techniques.
The paper tackles the problem of high prediction latency in deep neural networks by proposing a caching method for intermediate layer outputs, which reduces the effective number of layers by half for 91.58% of CIFAR-10 requests on ResNet-18.
Over the last few years, Deep Neural Networks (DNNs) have become ubiquitous owing to their high accuracy on real-world tasks. However, this increase in accuracy comes at the cost of computationally expensive models leading to higher prediction latencies. Prior efforts to reduce this latency such as quantization, model distillation, and any-time prediction models typically trade-off accuracy for performance. In this work, we observe that caching intermediate layer outputs can help us avoid running all the layers of a DNN for a sizeable fraction of inference requests. We find that this can potentially reduce the number of effective layers by half for 91.58% of CIFAR-10 requests run on ResNet-18. We present Freeze Inference, a system that introduces approximate caching at each intermediate layer and we discuss techniques to reduce the cache size and improve the cache hit rate. Finally, we discuss some of the open research challenges in realizing such a design.