Basic Reading Distillation
This addresses the computational inefficiency of LLMs for real-world deployment, offering a novel distillation approach that is incremental but distinct from existing methods.
The paper tackles the problem of deploying large language models (LLMs) in resource-constrained settings by proposing basic reading distillation (BRD), which trains small models to imitate LLMs' basic reading behaviors on generic texts, resulting in the small model outperforming or matching LLMs over 20x larger on various tasks.
Large language models (LLMs) have demonstrated remarkable abilities in various natural language processing areas, but they demand high computation resources which limits their deployment in real-world. Distillation is one technique to solve this problem through either knowledge distillation or task distillation. Both distillation approaches train small models to imitate specific features of LLMs, but they all neglect basic reading education for small models on generic texts that are \emph{unrelated} to downstream tasks. In this paper, we propose basic reading distillation (BRD) which educates a small model to imitate LLMs basic reading behaviors, such as named entity recognition, question raising and answering, on each sentence. After such basic education, we apply the small model on various tasks including language inference benchmarks and BIG-bench tasks. It shows that the small model can outperform or perform comparable to over 20x bigger LLMs. Analysis reveals that BRD effectively influences the probability distribution of the small model, and has orthogonality to either knowledge distillation or task distillation.