Churn Reduction via Distillation
This addresses the issue of maintaining stable predictions during model updates in real-world systems, which is incremental as it builds on existing distillation methods.
The paper tackled the problem of predictive churn in model updates by showing that distillation using the base model as a teacher is equivalent to training with an explicit churn constraint, and demonstrated that distillation performs strongly for low churn training across various datasets and architectures.
In real-world systems, models are frequently updated as more data becomes available, and in addition to achieving high accuracy, the goal is to also maintain a low difference in predictions compared to the base model (i.e. predictive "churn"). If model retraining results in vastly different behavior, then it could cause negative effects in downstream systems, especially if this churn can be avoided with limited impact on model accuracy. In this paper, we show an equivalence between training with distillation using the base model as the teacher and training with an explicit constraint on the predictive churn. We then show that distillation performs strongly for low churn training against a number of recent baselines on a wide range of datasets and model architectures, including fully-connected networks, convolutional networks, and transformers.