Regularizing Neural Networks with Meta-Learning Generative Models
This work addresses a key challenge in deep learning for small datasets by improving generative data augmentation, though it is incremental as it builds on existing methods with a novel regularization strategy.
The paper tackles the problem of generative data augmentation degrading accuracy due to uninformative synthetic samples by introducing meta generative regularization (MGR), which uses synthetic samples in a regularization term optimized via meta-learning, resulting in stable performance gains across six datasets, especially with smaller datasets.
This paper investigates methods for improving generative data augmentation for deep learning. Generative data augmentation leverages the synthetic samples produced by generative models as an additional dataset for classification with small dataset settings. A key challenge of generative data augmentation is that the synthetic data contain uninformative samples that degrade accuracy. This is because the synthetic samples do not perfectly represent class categories in real data and uniform sampling does not necessarily provide useful samples for tasks. In this paper, we present a novel strategy for generative data augmentation called meta generative regularization (MGR). To avoid the degradation of generative data augmentation, MGR utilizes synthetic samples in the regularization term for feature extractors instead of in the loss function, e.g., cross-entropy. These synthetic samples are dynamically determined to minimize the validation losses through meta-learning. We observed that MGR can avoid the performance degradation of naïve generative data augmentation and boost the baselines. Experiments on six datasets showed that MGR is effective particularly when datasets are smaller and stably outperforms baselines.