Can abstract concepts from LLM improve SLM performance?
This work addresses the problem of efficient AI deployment for resource-limited environments, but it is incremental as it builds on existing concept extraction techniques.
The paper tackled the challenge of deploying large language models on resource-constrained devices by transferring abstract concepts from LLMs to smaller models, resulting in a 7-15% accuracy improvement for Qwen3-0.6B through inference-time scaling.
Large language models (LLMs) excel at diverse tasks, but their deployment on resource-constrained devices remains challenging. Existing methods like quantization, pruning, and distillation can reduce memory footprint but often demand extensive experimentation and careful infrastructure design. Leveraging existing techniques for extracting high-level concepts (represented as steering vectors) from larger models, we investigate their transferability to smaller language models (SLM) during inference. We demonstrate through extensive experimentation that these concepts can be effectively transferred to smaller models, irrespective of their family (e.g., Phi, Llama, Qwen), leading to performance improvements across a wide range of tasks. Furthermore, we introduce inference-time scaling to enhance performance by dynamically adjusting the steering intensity which has resulted in a 7-15\% of accuracy improvement for Qwen3-0.6B.