Distillation of Large Language Models via Concrete Score Matching
This addresses the need for efficient deployment of large language models, offering a novel distillation method that is scalable and effective, though it is incremental as it builds on existing knowledge distillation techniques.
The paper tackles the problem of knowledge distillation for large language models by proposing Concrete Score Distillation (CSD), which overcomes limitations of existing methods like softmax smoothing and logit shift invariance, resulting in consistent performance improvements over recent distillation objectives and favorable fidelity-diversity trade-offs.
Large language models (LLMs) deliver remarkable performance but are costly to deploy, motivating knowledge distillation (KD) for efficient inference. Existing KD objectives typically match student and teacher probabilities via softmax, which blurs valuable logit information. While direct logit distillation (DLD) mitigates softmax smoothing, it fails to account for logit shift invariance, thereby restricting the solution space. We propose Concrete Score Distillation (CSD), a discrete score-matching objective that overcomes both softmax-induced smoothing and restrictions on the optimal solution set. We resolve the training instability and quadratic complexity of discrete score-matching in autoregressive LLMs, and the resulting CSD objective aligns relative logit differences across all vocabulary pairs between student and teacher with flexible weighting. We provide both mode-seeking and mode-covering instances within our framework and evaluate CSD on task-agnostic instruction-following and task-specific distillation using GPT-2-1.5B, OpenLLaMA-7B, and GEMMA-7B-IT. Experiments show that CSD consistently surpasses recent KD objectives, achieves favorable fidelity-diversity trade-offs, and yields complementary gains when combined with on-policy techniques, demonstrating its scalability and effectiveness for LLM distillation.