Adaptive Weighting in Knowledge Distillation: An Axiomatic Framework for Multi-Scale Teacher Ensemble Optimization
This provides a foundational framework for analyzing and improving knowledge distillation methods, addressing robustness and safety in AI applications, though it is incremental as it formalizes existing concepts rather than introducing a new paradigm.
The paper tackled the problem of heuristic weighting in multi-teacher knowledge distillation by developing an axiomatic framework for adaptive weighting across token, task, and context scales, establishing theoretical guarantees for existence, convergence, and robustness without relying on specific formulas.
Knowledge distillation with multiple teachers is increasingly used to improve robustness, efficiency, and safety, yet existing approaches rely largely on heuristic or implementation-specific weighting schemes. This paper develops an operator-agnostic axiomatic framework for adaptive weighting in multi-teacher knowledge distillation across three complementary scales: token, task, and context. We formalize structural conditions under which adaptive weighting operators are well-defined, admit multiple non-equivalent implementations, and can be hierarchically composed via product-structure normalization. Within this framework, we establish existence and non-uniqueness of conforming operators, characterize convergence of gradient-based optimization under standard assumptions, analyze stability and perturbation robustness, and provide an abstract formulation of safety-constrained distillation. The results decouple theoretical guarantees from specific weighting formulas, enabling principled analysis of adaptive distillation methods under heterogeneity, distribution shift, and safety constraints.