Synthetic Adaptive Guided Embeddings (SAGE): A Novel Knowledge Distillation Method
This work addresses efficient model compression for deployment in resource-constrained environments, representing an incremental improvement over existing distillation methods.
The paper tackles the problem of computational overhead and limited generalization in model distillation by proposing an adaptive distillation framework that dynamically augments training data in high-loss regions, achieving 91.2% on QNLI and 92.3% on SST-2 with a 66M-parameter student model.
Model distillation enables the transfer of knowledge from large-scale models to compact student models, facilitating deployment in resource-constrained environments. However, conventional distillation approaches often suffer from computational overhead and limited generalization. We propose a novel adaptive distillation framework that dynamically augments training data in regions of high student model loss. Using UMAP-based dimensionality reduction and nearest neighbor sampling, our method identifies underperforming regions in the embedding space and generates targeted synthetic examples to guide student learning. To further improve efficiency, we introduce a lightweight teacher-student interface that bypasses the teacher's input layer, enabling direct distillation on vectorized representations. Experiments across standard NLP benchmarks demonstrate that our 66M-parameter student model consistently matches or surpasses established baselines, achieving 91.2% on QNLI and 92.3% on SST-2, while training with fewer epochs. These results highlight the promise of loss-aware data augmentation and vectorized distillation for efficient and effective model compression.