Monitoring and Adapting ML Models on Mobile Devices
This addresses the challenge for ML operators in maintaining model performance on mobile devices, offering a novel system for automated monitoring and adaptation, though it is incremental in building on existing mobile ML deployment concepts.
The paper tackles the problem of tracking and maintaining ML model accuracy on mobile devices after deployment, where accuracy can degrade due to issues like data drift, and presents an end-to-end system that monitors and adapts models without user feedback, improving accuracy by an average of 15% on a driving car photo dataset.
ML models are increasingly being pushed to mobile devices, for low-latency inference and offline operation. However, once the models are deployed, it is hard for ML operators to track their accuracy, which can degrade unpredictably (e.g., due to data drift). We design the first end-to-end system for continuously monitoring and adapting models on mobile devices without requiring feedback from users. Our key observation is that often model degradation is due to a specific root cause, which may affect a large group of devices. Therefore, once the system detects a consistent degradation across a large number of devices, it employs a root cause analysis to determine the origin of the problem and applies a cause-specific adaptation. We evaluate the system on two computer vision datasets, and show it consistently boosts accuracy compared to existing approaches. On a dataset containing photos collected from driving cars, our system improves the accuracy on average by 15%.