Decoder-based Sense Knowledge Distillation
This work addresses the challenge of incorporating structured semantics into generative models for improved knowledge distillation, though it appears incremental as it extends prior encoder-based methods to decoders.
The paper tackles the problem of decoder-based large language models lacking structured lexical knowledge by introducing Decoder-based Sense Knowledge Distillation (DSKD), which integrates lexical resources during training without inference-time dictionary lookup, resulting in significant performance improvements on diverse benchmarks.
Large language models (LLMs) learn contextual embeddings that capture rich semantic information, yet they often overlook structured lexical knowledge such as word senses and relationships. Prior work has shown that incorporating sense dictionaries can improve knowledge distillation for encoder models, but their application to decoder as generative models remains challenging. In this paper, we introduce Decoder-based Sense Knowledge Distillation (DSKD), a framework that integrates lexical resources into the training of decoder-style LLMs without requiring dictionary lookup at inference time. Extensive experiments on diverse benchmarks demonstrate that DSKD significantly enhances knowledge distillation performance for decoders, enabling generative models to inherit structured semantics while maintaining efficient training.