MLDemon: Deployment Monitoring for Machine Learning Systems
This addresses the critical need for efficient post-deployment monitoring in ML systems, particularly for applications where data distributions evolve over time, offering a novel solution with theoretical guarantees.
The paper tackles the problem of monitoring machine learning models after deployment to ensure reliability under distribution shifts, proposing MLDemon which integrates unlabeled data and on-demand labels to estimate model performance in real-time, achieving superior results on temporal datasets with diverse drifts and proving minimax rate optimality.
Post-deployment monitoring of ML systems is critical for ensuring reliability, especially as new user inputs can differ from the training distribution. Here we propose a novel approach, MLDemon, for ML DEployment MONitoring. MLDemon integrates both unlabeled data and a small amount of on-demand labels to produce a real-time estimate of the ML model's current performance on a given data stream. Subject to budget constraints, MLDemon decides when to acquire additional, potentially costly, expert supervised labels to verify the model. On temporal datasets with diverse distribution drifts and models, MLDemon outperforms existing approaches. Moreover, we provide theoretical analysis to show that MLDemon is minimax rate optimal for a broad class of distribution drifts.