Soyoun Won

Soyoun Won, Sung-Ho Bae, Seong Tae Kim

Mixed sample data augmentation strategies are actively used when training deep neural networks (DNNs). Recent studies suggest that they are effective at various tasks. However, the impact of mixed sample data augmentation on model interpretability has not been widely studied. In this paper, we explore the relationship between model interpretability and mixed sample data augmentation, specifically in terms of feature attribution maps. To this end, we introduce a new metric that allows a comparison of model interpretability while minimizing the impact of occlusion robustness of the model. Experimental results show that several mixed sample data augmentation decreases the interpretability of the model and label mixing during data augmentation plays a significant role in this effect. This new finding suggests it is important to carefully adopt the mixed sample data augmentation method, particularly in applications where attribution map-based interpretability is important.

Aryan Yazdan Parast, Khawar Islam, Soyoun Won et al.

Deep neural networks often rely on spurious features to make predictions, which makes them brittle under distribution shift and on samples where the spurious correlation does not hold (e.g., minority-group examples). Recent studies have shown that, even in such settings, the feature extractor of an Empirical Risk Minimization (ERM)-trained model can learn rich and informative representations, and that much of the failure may be attributed to the classifier head. In particular, retraining a lightweight head while keeping the backbone frozen can substantially improve performance on shifted distributions and minority groups. Motivated by this observation, we propose a bilevel meta-learning method that performs augmentation directly in feature space to improve spurious correlation handling in the classifier head. Our method learns support-side feature edits such that, after a small number of inner-loop updates on the edited features, the classifier achieves lower loss on hard examples and improved worst-group performance. By operating at the backbone output rather than in pixel space or through end-to-end optimization, the method is highly efficient and stable, requiring only a few minutes of training on a single GPU. We further validate our method with CLIP-based visualizations, showing that the learned feature-space updates induce semantically meaningful shifts aligned with spurious attributes.