SoftNeuro: Fast Deep Inference using Multi-platform Optimization
This work addresses the need for faster and more efficient deep learning inference, which is crucial for reducing costs and environmental impact, but it appears incremental as it builds on existing optimization techniques.
The paper tackles the problem of accelerating deep learning inference on edge devices and servers by proposing SoftNeuro, a framework that separates algorithmic routines from network layers and uses dynamic programming to select the fastest execution path, achieving fast inference and efficient tuning.
Faster inference of deep learning models is highly demanded on edge devices and even servers, for both financial and environmental reasons. To address this issue, we propose SoftNeuro, a novel, high-performance inference framework with efficient performance tuning. The key idea is to separate algorithmic routines from network layers. Our framework maximizes the inference performance by profiling various routines for each layer and selecting the fastest path. To efficiently find the best path, we propose a routine-selection algorithm based on dynamic programming. Experiments show that the proposed framework achieves both fast inference and efficient tuning.