Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model
This work addresses a practical limitation in real-world applications where access to teacher parameters or training data is restricted due to privacy or storage issues, offering a novel approach for model compression under such constraints.
The paper tackles the problem of knowledge distillation when the teacher model is a black-box that only provides class decisions, not softmax outputs, by constructing soft labels based on distances to decision boundaries and generating pseudo samples when training data is unavailable. The results show effectiveness across various benchmark networks and datasets, with code made publicly available.
Knowledge distillation (KD) is a successful approach for deep neural network acceleration, with which a compact network (student) is trained by mimicking the softmax output of a pre-trained high-capacity network (teacher). In tradition, KD usually relies on access to the training samples and the parameters of the white-box teacher to acquire the transferred knowledge. However, these prerequisites are not always realistic due to storage costs or privacy issues in real-world applications. Here we propose the concept of decision-based black-box (DB3) knowledge distillation, with which the student is trained by distilling the knowledge from a black-box teacher (parameters are not accessible) that only returns classes rather than softmax outputs. We start with the scenario when the training set is accessible. We represent a sample's robustness against other classes by computing its distances to the teacher's decision boundaries and use it to construct the soft label for each training sample. After that, the student can be trained via standard KD. We then extend this approach to a more challenging scenario in which even accessing the training data is not feasible. We propose to generate pseudo samples distinguished by the teacher's decision boundaries to the largest extent and construct soft labels for them, which are used as the transfer set. We evaluate our approaches on various benchmark networks and datasets and experiment results demonstrate their effectiveness. Codes are available at: https://github.com/zwang84/zsdb3kd.