Failure-Resilient Distributed Inference with Model Compression over Heterogeneous Edge Devices
This addresses the problem of reliable and efficient inference for IoT applications, but it is incremental as it builds on existing distributed inference and knowledge distillation techniques.
The paper tackles the challenge of performing distributed deep learning inference on heterogeneous and failure-prone IoT devices by introducing RoCoIn, a mechanism that uses knowledge distillation to create compact student models and strategically groups devices for redundancy, achieving improved response latency and failure resilience compared to baselines.
The distributed inference paradigm enables the computation workload to be distributed across multiple devices, facilitating the implementations of deep learning based intelligent services on extremely resource-constrained Internet of Things (IoT) scenarios. Yet it raises great challenges to perform complicated inference tasks relying on a cluster of IoT devices that are heterogeneous in their computing/communication capacity and prone to crash or timeout failures. In this paper, we present RoCoIn, a robust cooperative inference mechanism for locally distributed execution of deep neural network-based inference tasks over heterogeneous edge devices. It creates a set of independent and compact student models that are learned from a large model using knowledge distillation for distributed deployment. In particular, the devices are strategically grouped to redundantly deploy and execute the same student model such that the inference process is resilient to any local failures, while a joint knowledge partition and student model assignment scheme are designed to minimize the response latency of the distributed inference system in the presence of devices with diverse capacities. Extensive simulations are conducted to corroborate the superior performance of our RoCoIn for distributed inference compared to several baselines, and the results demonstrate its efficacy in timely inference and failure resiliency.