Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks
This work addresses the computational inefficiency of LLMs for real-world applications, though it is incremental as it applies existing methods to new data.
This study tackled the problem of compressing Large Language Models (LLMs) for deployment in resource-constrained environments by using Knowledge Distillation on Question Answering tasks, achieving student models that retain over 90% of teacher performance while reducing parameters by up to 57.1%.
Large Language Models (LLMs) have demonstrated outstanding performance across a range of NLP tasks, however, their computational demands hinder their deployment in real-world, resource-constrained environments. This work investigates the extent to which LLMs can be compressed using Knowledge Distillation (KD) while maintaining strong performance on Question Answering (QA) tasks. We evaluate student models distilled from the Pythia and Qwen2.5 families on two QA benchmarks, SQuAD and MLQA, under zero-shot and one-shot prompting conditions. Results show that student models retain over 90% of their teacher models' performance while reducing parameter counts by up to 57.1%. Furthermore, one-shot prompting yields additional performance gains over zero-shot setups for both model families. These findings underscore the trade-off between model efficiency and task performance, demonstrating that KD, combined with minimal prompting, can yield compact yet capable QA systems suitable for resource-constrained applications.