GhostNetV3-Small: A Tailored Architecture and Comparative Study of Distillation Strategies for Tiny Images
This work addresses model compression for resource-constrained edge devices, but it is incremental as it adapts an existing architecture and evaluates standard distillation methods.
The paper tackled the problem of adapting deep neural networks for efficient inference on edge devices by proposing GhostNetV3-Small, a modified architecture tailored for low-resolution images like CIFAR-10, which achieved 93.94% accuracy, outperforming the original GhostNetV3, while finding that knowledge distillation strategies reduced accuracy compared to baseline training.
Deep neural networks have achieved remarkable success across a range of tasks, however their computational demands often make them unsuitable for deployment on resource-constrained edge devices. This paper explores strategies for compressing and adapting models to enable efficient inference in such environments. We focus on GhostNetV3, a state-of-the-art architecture for mobile applications, and propose GhostNetV3-Small, a modified variant designed to perform better on low-resolution inputs such as those in the CIFAR-10 dataset. In addition to architectural adaptation, we provide a comparative evaluation of knowledge distillation techniques, including traditional knowledge distillation, teacher assistants, and teacher ensembles. Experimental results show that GhostNetV3-Small significantly outperforms the original GhostNetV3 on CIFAR-10, achieving an accuracy of 93.94%. Contrary to expectations, all examined distillation strategies led to reduced accuracy compared to baseline training. These findings indicate that architectural adaptation can be more impactful than distillation in small-scale image classification tasks, highlighting the need for further research on effective model design and advanced distillation techniques for low-resolution domains.