Scaling Laws for Task-Specific LLM Distillation
For practitioners deploying LLMs in latency/cost-constrained domains, this provides a reusable framework to make compression decisions based on scaling laws.
This paper derives empirical scaling laws for domain-specific LLM compression in quantitative finance, showing that in-domain task quality degrades predictably under compression while general knowledge collapses earlier, and that chain-of-thought supervision can recover erased general knowledge.
Large Language Models (LLMs) achieve strong performance across a growing range of domains, yet their scale poses deployment challenges in applications where latency and cost constraints are critical. This paper derives empirical scaling laws for domain-specific LLM compression, quantifying how in-domain and general knowledge performance scale with dataset size, compression ratio, supervision format, and iterative pruning schedule. Using quantitative finance as our application domain, we compare logit-based and LoRA-based distillation under iterative structural pruning, introducing a blended chain-of-thought supervision loss that stabilizes KL-divergence distillation over reasoning traces. In-domain task quality degrades predictably under compression while general-knowledge benchmarks collapse well before the same point; supervision format is the key driver of this tradeoff, with chain-of-thought supervision actively recovering general knowledge that pruning erases. We release the headline dataset FinHeadlineMix, scaling law results, and practical recommendations to provide a reusable framework for domain-specific compression decisions.