SysNoise: Exploring and Benchmarking Training-Deployment System Inconsistency
This addresses a critical but overlooked problem in deep learning deployment for practitioners, as it highlights how system mismatches can degrade model performance, though it is incremental in benchmarking an existing issue.
The paper introduces SysNoise, a noise caused by system inconsistencies between training and deployment environments, and benchmarks its impact on over 20 models across tasks like image classification and NLP, revealing significant robustness issues with limited mitigation effects.
Extensive studies have shown that deep learning models are vulnerable to adversarial and natural noises, yet little is known about model robustness on noises caused by different system implementations. In this paper, we for the first time introduce SysNoise, a frequently occurred but often overlooked noise in the deep learning training-deployment cycle. In particular, SysNoise happens when the source training system switches to a disparate target system in deployments, where various tiny system mismatch adds up to a non-negligible difference. We first identify and classify SysNoise into three categories based on the inference stage; we then build a holistic benchmark to quantitatively measure the impact of SysNoise on 20+ models, comprehending image classification, object detection, instance segmentation and natural language processing tasks. Our extensive experiments revealed that SysNoise could bring certain impacts on model robustness across different tasks and common mitigations like data augmentation and adversarial training show limited effects on it. Together, our findings open a new research topic and we hope this work will raise research attention to deep learning deployment systems accounting for model performance. We have open-sourced the benchmark and framework at https://modeltc.github.io/systemnoise_web.