Addressing Discrepancies in Semantic and Visual Alignment in Neural Networks
This addresses a specific issue in image classification for improving model robustness, but it is incremental as it builds on existing diffusion methods.
The paper tackles the problem of misalignment between semantic and visual similarity in neural networks for image classification by proposing a data augmentation technique using diffusion-based semantic mixing to generate hybrids of classes. Results show an increase in semantic alignment, as measured by improved performance on adversarially perturbed data.
For the task of image classification, neural networks primarily rely on visual patterns. In robust networks, we would expect for visually similar classes to be represented similarly. We consider the problem of when semantically similar classes are visually dissimilar, and when visual similarity is present among non-similar classes. We propose a data augmentation technique with the goal of better aligning semantically similar classes with arbitrary (non-visual) semantic relationships. We leverage recent work in diffusion-based semantic mixing to generate semantic hybrids of two classes, and these hybrids are added to the training set as augmented data. We evaluate whether the method increases semantic alignment by evaluating model performance on adversarially perturbed data, with the idea that it should be easier for an adversary to switch one class to a similarly represented class. Results demonstrate that there is an increase in alignment of semantically similar classes when using our proposed data augmentation method.