A Framework for Monitoring and Retraining Language Models in Real-World Applications
This addresses the need for efficient model maintenance post-deployment in real-world AI systems, but it is incremental as it builds on existing monitoring and retraining concepts.
The paper tackles the problem of continuous monitoring and retraining of language models in real-world applications, examining how retraining decisions affect model performance and resource utilization, with results focused on multilabel classification models.
In the Machine Learning (ML) model development lifecycle, training candidate models using an offline holdout dataset and identifying the best model for the given task is only the first step. After the deployment of the selected model, continuous model monitoring and model retraining is required in many real-world applications. There are multiple reasons for retraining, including data or concept drift, which may be reflected on the model performance as monitored by an appropriate metric. Another motivation for retraining is the acquisition of increasing amounts of data over time, which may be used to retrain and improve the model performance even in the absence of drifts. We examine the impact of various retraining decision points on crucial factors, such as model performance and resource utilization, in the context of Multilabel Classification models. We explain our key decision points and propose a reference framework for designing an effective model retraining strategy.