A Mathematics Framework of Artificial Shifted Population Risk and Its Further Understanding Related to Consistency Regularization
This work provides a theoretical foundation for data augmentation in deep learning, which is incremental but addresses a gap in understanding for researchers and practitioners.
The paper tackles the lack of theoretical understanding of the relationship between data augmentation and consistency regularization by introducing a mathematical framework that shows the expected risk of a shifted population equals the original risk plus a gap term interpreted as consistency regularization, and experiments demonstrate their method outperforms others in generalization and convergence across various scenarios.
Data augmentation is an important technique in training deep neural networks as it enhances their ability to generalize and remain robust. While data augmentation is commonly used to expand the sample size and act as a consistency regularization term, there is a lack of research on the relationship between them. To address this gap, this paper introduces a more comprehensive mathematical framework for data augmentation. Through this framework, we establish that the expected risk of the shifted population is the sum of the original population risk and a gap term, which can be interpreted as a consistency regularization term. The paper also provides a theoretical understanding of this gap, highlighting its negative effects on the early stages of training. We also propose a method to mitigate these effects. To validate our approach, we conducted experiments using same data augmentation techniques and computing resources under several scenarios, including standard training, out-of-distribution, and imbalanced classification. The results demonstrate that our methods surpass compared methods under all scenarios in terms of generalization ability and convergence stability. We provide our code implementation at the following link: https://github.com/ydlsfhll/ASPR.