Split Knowledge Distillation for Large Models in IoT: Architecture, Challenges, and Solutions
This addresses data privacy and resource constraints for IoT applications like voice assistants and healthcare, but it appears incremental as it builds on existing knowledge distillation and split learning techniques.
The paper tackles the challenge of deploying large models in IoT systems by proposing a split knowledge distillation framework to distill them into smaller versions for devices, with a case study evaluating its feasibility and performance.
Large models (LMs) have immense potential in Internet of Things (IoT) systems, enabling applications such as intelligent voice assistants, predictive maintenance, and healthcare monitoring. However, training LMs on edge servers raises data privacy concerns, while deploying them directly on IoT devices is constrained by limited computational and memory resources. We analyze the key challenges of training LMs in IoT systems, including energy constraints, latency requirements, and device heterogeneity, and propose potential solutions such as dynamic resource management, adaptive model partitioning, and clustered collaborative training. Furthermore, we propose a split knowledge distillation framework to efficiently distill LMs into smaller, deployable versions for IoT devices while ensuring raw data remains local. This framework integrates knowledge distillation and split learning to minimize energy consumption and meet low model training delay requirements. A case study is presented to evaluate the feasibility and performance of the proposed framework.