Evaluation-Strategy Gap in Fault Diagnosis of Deep Learning Programs
For researchers and practitioners developing fault diagnosis tools for deep learning, this work highlights the inadequacy of within-program evaluation and provides insights into feature generalization.
The paper identifies a 0.190 gap in balanced accuracy for fault diagnosis techniques in deep learning programs when evaluated on unseen programs versus within-program cross-validation, and finds that curvature features generalize better than optimizer and activation features.
Deep Learning (DL) programs can fail during training for many reasons, and diagnosing the cause is a costly and time-consuming maintenance task. Techniques for diagnosing such failures are commonly assessed using within-program cross-validation, which may be inadequate for deployment settings involving previously unseen programs. It is therefore necessary to assess how performance differs across these settings and to identify the causes of any performance gap in established fault diagnosis techniques for DL. We investigate this gap using DynFault, a corpus of 5,542 fault-injected training traces from 38 real-world DL programs. We found a gap of 0.190 in balanced accuracy for existing fault diagnosis techniques between within-program evaluation and holding out whole programs. We also found the gap comes from program-level structure in the features, which led us to examine two runtime feature sets, curvature features and optimizer features, and their behavior on unseen programs. We found that curvature features are useful for instability detection on unseen programs, while optimizer and activation features help only on programs seen during training.