QUADS: QUAntized Distillation Framework for Efficient Speech Language Understanding
This provides an efficient solution for real-world SLU applications in resource-constrained settings, though it appears incremental as it builds on existing distillation and quantization techniques.
The paper tackles the problem of balancing performance and efficiency in Spoken Language Understanding (SLU) systems for resource-constrained environments by proposing QUADS, a unified framework that combines distillation and quantization. It achieves 71.13% accuracy on SLURP and 99.20% on FSC with minor degradations up to 5.56%, while reducing computational complexity by 60-73× and model size by 83-700×.
Spoken Language Understanding (SLU) systems must balance performance and efficiency, particularly in resource-constrained environments. Existing methods apply distillation and quantization separately, leading to suboptimal compression as distillation ignores quantization constraints. We propose QUADS, a unified framework that optimizes both through multi-stage training with a pre-tuned model, enhancing adaptability to low-bit regimes while maintaining accuracy. QUADS achieves 71.13\% accuracy on SLURP and 99.20\% on FSC, with only minor degradations of up to 5.56\% compared to state-of-the-art models. Additionally, it reduces computational complexity by 60--73$\times$ (GMACs) and model size by 83--700$\times$, demonstrating strong robustness under extreme quantization. These results establish QUADS as a highly efficient solution for real-world, resource-constrained SLU applications.