Assessing Data Augmentation-Induced Bias in Training and Testing of Machine Learning Models
This addresses bias issues for practitioners in software engineering using data augmentation, but it is incremental as it builds on existing techniques without introducing a new method.
The paper tackled the problem of bias induced by data augmentation in machine learning models, particularly in software engineering domains like flaky test classification, and demonstrated through a case study that including augmented samples in testing sets can impact model evaluation.
Data augmentation has become a standard practice in software engineering to address limited or imbalanced data sets, particularly in specialized domains like test classification and bug detection where data can be scarce. Although techniques such as SMOTE and mutation-based augmentation are widely used in software testing and debugging applications, a rigorous understanding of how augmented training data impacts model bias is lacking. It is especially critical to consider bias in scenarios where augmented data sets are used not just in training but also in testing models. Through a comprehensive case study of flaky test classification, we demonstrate how to test for bias and understand the impact that the inclusion of augmented samples in testing sets can have on model evaluation.