Cooperative Knowledge Distillation: A Learner Agnostic Approach
This addresses the problem of inefficient knowledge transfer in machine learning for practitioners, offering a more flexible and effective method, though it is incremental as it builds on existing distillation concepts.
The paper tackles the limitations of traditional knowledge distillation by proposing cooperative distillation, where models act as both students and teachers to transfer knowledge based on specific deficiencies, using counterfactual instance generation. The result shows that this approach outperforms baselines like transfer learning and other distillation algorithms on several datasets and works in settings where existing techniques fail.
Knowledge distillation is a simple but powerful way to transfer knowledge between a teacher model to a student model. Existing work suffers from at least one of the following key limitations in terms of direction and scope of transfer which restrict its use: all knowledge is transferred from teacher to student regardless of whether or not that knowledge is useful, the student is the only one learning in this exchange, and typically distillation transfers knowledge only from a single teacher to a single student. We formulate a novel form of knowledge distillation in which many models can act as both students and teachers which we call cooperative distillation. The models cooperate as follows: a model (the student) identifies specific deficiencies in it's performance and searches for another model (the teacher) who encodes learned knowledge into instructional virtual instances via counterfactual instance generation. Because different models may have different strengths and weaknesses, all models can act as either students or teachers (cooperation) when appropriate and only distill knowledge in areas specific to their strengths (focus). Since counterfactuals as a paradigm are not tied to any specific algorithm, we can use this method to distill knowledge between learners of different architectures, algorithms, and even feature spaces. We demonstrate that our approach not only outperforms baselines such as transfer learning, self-supervised learning, and multiple knowledge distillation algorithms on several datasets, but it can also be used in settings where the aforementioned techniques cannot.