Poster: Self-Supervised Quantization-Aware Knowledge Distillation
This work addresses the need for more efficient and accessible quantization techniques in machine learning, particularly for resource-constrained applications, though it appears incremental as it builds on existing QAT approaches.
The paper tackles the problem of accuracy loss and label dependency in quantization-aware training (QAT) by proposing a self-supervised framework called SQAKD, which unifies quantization dynamics and co-optimizes KL-Loss and discretization error, resulting in significant performance improvements over existing QAT methods.
Quantization-aware training (QAT) starts with a pre-trained full-precision model and performs quantization during retraining. However, existing QAT works require supervision from the labels and they suffer from accuracy loss due to reduced precision. To address these limitations, this paper proposes a novel Self-Supervised Quantization-Aware Knowledge Distillation framework (SQAKD). SQAKD first unifies the forward and backward dynamics of various quantization functions and then reframes QAT as a co-optimization problem that simultaneously minimizes the KL-Loss and the discretization error, in a self-supervised manner. The evaluation shows that SQAKD significantly improves the performance of various state-of-the-art QAT works. SQAKD establishes stronger baselines and does not require extensive labeled training data, potentially making state-of-the-art QAT research more accessible.