Efficient Few-Shot Learning for Edge AI via Knowledge Distillation on MobileViT
This work addresses the need for efficient and adaptable deep learning models in edge AI scenarios, such as low-data regimes and resource-constrained environments, though it is incremental as it builds on existing knowledge distillation and MobileViT techniques.
The paper tackled the problem of deploying few-shot learning models on edge devices by proposing a knowledge distillation method for MobileViT, achieving accuracy improvements of 14% (one-shot) and 6.7% (five-shot) on MiniImageNet while reducing parameters by 69% and computational complexity by 88%.
Efficient and adaptable deep learning models are an important area of deep learning research, driven by the need for highly efficient models on edge devices. Few-shot learning enables the use of deep learning models in low-data regimes, a capability that is highly sought after in real-world applications where collecting large annotated datasets is costly or impractical. This challenge is particularly relevant in edge scenarios, where connectivity may be limited, low-latency responses are required, or energy consumption constraints are critical. We propose and evaluate a pre-training method for the MobileViT backbone designed for edge computing. Specifically, we employ knowledge distillation, which transfers the generalization ability of a large-scale teacher model to a lightweight student model. This method achieves accuracy improvements of 14% and 6.7% for one-shot and five-shot classification, respectively, on the MiniImageNet benchmark, compared to the ResNet12 baseline, while reducing by 69% the number of parameters and by 88% the computational complexity of the model, in FLOPs. Furthermore, we deployed the proposed models on a Jetson Orin Nano platform and measured power consumption directly at the power supply, showing that the dynamic energy consumption is reduced by 37% with a latency of 2.6 ms. These results demonstrate that the proposed method is a promising and practical solution for deploying few-shot learning models on edge AI hardware.