Beyond Student: An Asymmetric Network for Neural Network Inheritance
This work addresses model compression for efficient deployment, offering a novel approach beyond traditional distillation, though it appears incremental as it builds on existing KD and decomposition techniques.
The paper tackled the performance limitations of student networks in Knowledge Distillation due to capacity gaps by proposing InherNet, an asymmetric low-rank decomposition method that reconstructs a lightweight network from teacher weights, achieving higher performance than student networks of similar sizes in unimodal and multimodal tasks.
Knowledge Distillation (KD) has emerged as a powerful technique for model compression, enabling lightweight student networks to benefit from the performance of redundant teacher networks. However, the inherent capacity gap often limits the performance of student networks. Inspired by the expressiveness of pretrained teacher networks, a compelling research question arises: is there a type of network that can not only inherit the teacher's structure but also maximize the inheritance of its knowledge? Furthermore, how does the performance of such an inheriting network compare to that of student networks, all benefiting from the same teacher network? To further explore this question, we propose InherNet, a neural network inheritance method that performs asymmetric low-rank decomposition on the teacher's weights and reconstructs a lightweight yet expressive network without significant architectural disruption. By leveraging Singular Value Decomposition (SVD) for initialization to ensure the inheritance of principal knowledge, InherNet effectively balances depth, width, and compression efficiency. Experimental results across unimodal and multimodal tasks demonstrate that InherNet achieves higher performance compared to student networks of similar parameter sizes. Our findings reveal a promising direction for future research in efficient model compression beyond traditional distillation.