Boosting classification reliability of NLP transformer models in the long run
This addresses the challenge of model reliability over time for NLP practitioners, but it is incremental as it builds on existing fine-tuning methods.
The paper tackles the problem of maintaining classification performance of BERT models over time in NLP tasks, finding that fine-tuning with all available unlabeled data from new periods is the best approach, though it only slows performance decline without preventing it, with results based on over 8 million COVID-19 vaccination comments in Hungary.
Transformer-based machine learning models have become an essential tool for many natural language processing (NLP) tasks since the introduction of the method. A common objective of these projects is to classify text data. Classification models are often extended to a different topic and/or time period. In these situations, deciding how long a classification is suitable for and when it is worth re-training our model is difficult. This paper compares different approaches to fine-tune a BERT model for a long-running classification task. We use data from different periods to fine-tune our original BERT model, and we also measure how a second round of annotation could boost the classification quality. Our corpus contains over 8 million comments on COVID-19 vaccination in Hungary posted between September 2020 and December 2021. Our results show that the best solution is using all available unlabeled comments to fine-tune a model. It is not advisable to focus only on comments containing words that our model has not encountered before; a more efficient solution is randomly sample comments from the new period. Fine-tuning does not prevent the model from losing performance but merely slows it down. In a rapidly changing linguistic environment, it is not possible to maintain model performance without regularly annotating new text.