Investigating Generalization in Neural Networks under Optimally Evolved Training Perturbations
This work addresses the problem of generalization robustness in neural networks for researchers and practitioners, though it is incremental as it builds on existing data distribution shift methods.
The paper investigates how neural networks generalize when training data is minimally corrupted by pixel perturbations, finding that such corruption can cause severe overfitting, and proposes an evolutionary algorithm that outperforms previous methods on state-of-the-art CNNs, with SGD showing more resilience than adaptive optimizers.
In this paper, we study the generalization properties of neural networks under input perturbations and show that minimal training data corruption by a few pixel modifications can cause drastic overfitting. We propose an evolutionary algorithm to search for optimal pixel perturbations using novel cost function inspired from literature in domain adaptation that explicitly maximizes the generalization gap and domain divergence between clean and corrupted images. Our method outperforms previous pixel-based data distribution shift methods on state-of-the-art Convolutional Neural Networks (CNNs) architectures. Interestingly, we find that the choice of optimization plays an important role in generalization robustness due to the empirical observation that SGD is resilient to such training data corruption unlike adaptive optimization techniques (ADAM). Our source code is available at https://github.com/subhajitchaudhury/evo-shift.