Automated Heterogeneous Low-Bit Quantization of Multi-Model Deep Learning Inference Pipeline
This addresses the problem of efficient edge deployment for multi-model deep learning systems, but appears incremental as it builds on existing quantization methods.
The paper tackles the challenge of deploying multiple deep neural networks in inference pipelines on edge devices by introducing an automated heterogeneous quantization approach to balance accuracy and latency, though no concrete numbers are provided.
Multiple Deep Neural Networks (DNNs) integrated into single Deep Learning (DL) inference pipelines e.g. Multi-Task Learning (MTL) or Ensemble Learning (EL), etc., albeit very accurate, pose challenges for edge deployment. In these systems, models vary in their quantization tolerance and resource demands, requiring meticulous tuning for accuracy-latency balance. This paper introduces an automated heterogeneous quantization approach for DL inference pipelines with multiple DNNs.