Unravelling Small Sample Size Problems in the Deep Learning World
This addresses the challenge of applying deep learning to domains with limited training data, which is an incremental improvement for specialized solutions in S^3 problems.
The paper tackles the problem of deep learning models performing poorly on small sample size (S^3) problems by reviewing existing algorithms and proposing a Dynamic Attention Pooling approach to extract global information from discriminative feature map parts, achieving competitive results on datasets like SVHN, C10, C100, and TinyImageNet.
The growth and success of deep learning approaches can be attributed to two major factors: availability of hardware resources and availability of large number of training samples. For problems with large training databases, deep learning models have achieved superlative performances. However, there are a lot of \textit{small sample size or $S^3$} problems for which it is not feasible to collect large training databases. It has been observed that deep learning models do not generalize well on $S^3$ problems and specialized solutions are required. In this paper, we first present a review of deep learning algorithms for small sample size problems in which the algorithms are segregated according to the space in which they operate, i.e. input space, model space, and feature space. Secondly, we present Dynamic Attention Pooling approach which focuses on extracting global information from the most discriminative sub-part of the feature map. The performance of the proposed dynamic attention pooling is analyzed with state-of-the-art ResNet model on relatively small publicly available datasets such as SVHN, C10, C100, and TinyImageNet.