Practical Insights of Repairing Model Problems on Image Classification
This work addresses model degradation issues for practitioners in industrial settings, but it is incremental as it provides insights from comparing existing methods.
The paper tackles the problem of model degradation in image classification, where additional training can turn previously correct predictions into errors, and finds that practitioners must continuously balance accuracy with degradation prevention based on dataset availability and system lifecycle.
Additional training of a deep learning model can cause negative effects on the results, turning an initially positive sample into a negative one (degradation). Such degradation is possible in real-world use cases due to the diversity of sample characteristics. That is, a set of samples is a mixture of critical ones which should not be missed and less important ones. Therefore, we cannot understand the performance by accuracy alone. While existing research aims to prevent a model degradation, insights into the related methods are needed to grasp their benefits and limitations. In this talk, we will present implications derived from a comparison of methods for reducing degradation. Especially, we formulated use cases for industrial settings in terms of arrangements of a data set. The results imply that a practitioner should care about better method continuously considering dataset availability and life cycle of an AI system because of a trade-off between accuracy and preventing degradation.