A monitoring framework for deployed machine learning models with supply chain examples
This work addresses the challenge of ensuring prediction quality and detecting issues in deployed ML models, especially in resource-intensive big data settings, but it is incremental as it applies existing drift detection methods to a new application area.
The paper tackles the problem of monitoring deployed machine learning models in production, particularly in big data environments like supply chains, by proposing a framework and implementing it to study drift in features, predictions, and performance on three real datasets, finding that while features and predictions showed statistically significant drifts, model performance remained stable.
Actively monitoring machine learning models during production operations helps ensure prediction quality and detection and remediation of unexpected or undesired conditions. Monitoring models already deployed in big data environments brings the additional challenges of adding monitoring in parallel to the existing modelling workflow and controlling resource requirements. In this paper, we describe (1) a framework for monitoring machine learning models; and, (2) its implementation for a big data supply chain application. We use our implementation to study drift in model features, predictions, and performance on three real data sets. We compare hypothesis test and information theoretic approaches to drift detection in features and predictions using the Kolmogorov-Smirnov distance and Bhattacharyya coefficient. Results showed that model performance was stable over the evaluation period. Features and predictions showed statistically significant drifts; however, these drifts were not linked to changes in model performance during the time of our study.