Empirical Evaluation of Knowledge Distillation from Transformers to Subquadratic Language Models
This work tackles the efficiency problem for deploying large language models by exploring distillation to subquadratic architectures, but it is incremental as it focuses on empirical evaluation of existing methods.
The study systematically evaluated knowledge distillation from a Transformer teacher to eight subquadratic student models, such as SSMs and linear attention, to address the quadratic complexity bottleneck in inference, finding insights into trade-offs between efficiency and performance on NLP benchmarks.
Knowledge distillation is a widely used technique for compressing large language models (LLMs), in which a smaller student model is trained to mimic a larger teacher model. Typically, both the teacher and student models are Transformer-based architectures, leveraging softmax attention for sequence modeling. However, the quadratic complexity of self-attention during inference remains a significant bottleneck, motivating the exploration of subquadratic alternatives such as structured state-space models (SSMs), linear attention, and recurrent architectures. In this work, we systematically evaluate the transferability of knowledge distillation from a Transformer teacher model to eight subquadratic student architectures. Our study investigates which subquadratic model can most effectively approximate the teacher model's learned representations through knowledge distillation, and how different architectural design choices influence the training dynamics. We further investigate the impact of initialization strategies, such as matrix mixing and query-key-value (QKV) copying, on the adaptation process. Our empirical results on multiple NLP benchmarks provide insights into the trade-offs between efficiency and performance, highlighting key factors for successful knowledge transfer to subquadratic architectures.