Sample-based Regularization: A Transfer Learning Strategy Toward Better Generalization
This work addresses the challenge of limited data in transfer learning for practitioners, offering an incremental improvement over prior methods.
The paper tackled the problem of overfitting in deep neural networks when training with small datasets by proposing sample-based regularization (SBR), a method that avoids using source knowledge during training, and experimental results showed it outperformed existing methods in various configurations.
Training a deep neural network with a small amount of data is a challenging problem as it is vulnerable to overfitting. However, one of the practical difficulties that we often face is to collect many samples. Transfer learning is a cost-effective solution to this problem. By using the source model trained with a large-scale dataset, the target model can alleviate the overfitting originated from the lack of training data. Resorting to the ability of generalization of the source model, several methods proposed to use the source knowledge during the whole training procedure. However, this is likely to restrict the potential of the target model and some transferred knowledge from the source can interfere with the training procedure. For improving the generalization performance of the target model with a few training samples, we proposed a regularization method called sample-based regularization (SBR), which does not rely on the source's knowledge during training. With SBR, we suggested a new training framework for transfer learning. Experimental results showed that our framework outperformed existing methods in various configurations.