Post-Training Probability Manifold Correction via Structured SVD Pruning and Self-Referential Distillation
This addresses the cost and efficiency challenges for deploying large language models, though it is incremental as it builds on existing pruning and distillation techniques.
The paper tackles the problem of expensive deployment of large language models by introducing SparseKD, a post-training compression method that combines structured SVD pruning with self-referential knowledge distillation, achieving 15-65% parameter reduction with acceptable quality trade-offs and improving model quality by 39% relative to the original checkpoint.
Large language models are expensive to deploy. We introduce Sparse Knowledge Distillation (SparseKD), a post-training method that compresses transformer models by combining structured SVD pruning with self-referential knowledge distillation. The key insight is simple: instead of using an external teacher, the model teaches itself by matching its own probability distribution from before compression. This self-referential setup enables surprisingly strong quality recovery after aggressive pruning. Our experiments reveal an unexpected finding: self-referential distillation alone, applied post-training under an identical objective and fixed calibration dataset, improves model quality by 39% relative to the original converged checkpoint. When combined with structured pruning, SparseKD achieves 15-65% parameter reduction with acceptable quality trade-offs. Kernel profiling shows that speedups arise entirely from reduced dense matrix multiplication in feed-forward layers while attention remains unchanged, making this approach complementary to attention optimizations. We validate across two model families (0.6B and 3.8B parameters) with multi-seed experiments confirming high reproducibility. SparseKD requires no external super-teacher, no architectural changes, and no custom inference kernels, making it immediately deployable with existing infrastructure.