Confidence Preservation Property in Knowledge Distillation Abstractions
This work addresses the need for more reliable and efficient models in social media content moderation by focusing on an incremental improvement in knowledge distillation techniques.
The paper investigates whether distilled TinyBERT models preserve the confidence values of the original BERT models and explores how this confidence preservation property can guide hyperparameter tuning in the distillation process, finding that it improves model reliability without specifying concrete numerical results.
Social media platforms prevent malicious activities by detecting harmful content of posts and comments. To that end, they employ large-scale deep neural network language models for sentiment analysis and content understanding. Some models, like BERT, are complex, and have numerous parameters, which makes them expensive to operate and maintain. To overcome these deficiencies, industry experts employ a knowledge distillation compression technique, where a distilled model is trained to reproduce the classification behavior of the original model. The distillation processes terminates when the distillation loss function reaches the stopping criteria. This function is mainly designed to ensure that the original and the distilled models exhibit alike classification behaviors. However, besides classification accuracy, there are additional properties of the original model that the distilled model should preserve to be considered as an appropriate abstraction. In this work, we explore whether distilled TinyBERT models preserve confidence values of the original BERT models, and investigate how this confidence preservation property could guide tuning hyperparameters of the distillation process.