Data Diversity as Implicit Regularization: How Does Diversity Shape the Weight Space of Deep Neural Networks?
This work addresses the unknown mechanism of data diversity's impact on model improvements, providing insights for researchers in deep learning regularization.
The paper investigates how data diversity from augmentation affects deep neural network weight spaces using Random Matrix Theory, revealing that diversity alters weight spectral distributions similarly to regularization techniques like dropout and more closely than weight decay.
Data augmentation that introduces diversity into the input data has long been used in training deep learning models. It has demonstrated benefits in improving robustness and generalization, practically aligning well with other regularization strategies such as dropout and weight decay. However, the underlying mechanism of how diverse training data contributes to model improvements remains unknown. In this paper, we investigate the impact of data diversity on the weight space of deep neural networks using Random Matrix Theory. Through spectral analysis and comparing models trained with data augmentation, dropout, and weight decay, we reveal that increasing data diversity alters the weight spectral distribution similarly to other regularization techniques, while displaying a pattern more closely aligned with dropout than with weight decay. Building on these insights, we propose a metric to explain and compare the benefits of diversity introduced by traditional data augmentations and those achieved through synthetic data.