On the Transferability of Winning Tickets in Non-Natural Image Datasets
This addresses the problem of data scarcity in non-natural domains for researchers, showing transfer benefits but highlighting limitations, making it incremental by extending lottery ticket hypothesis to new data types.
The paper investigates whether pruned neural networks (winning tickets) from natural image datasets generalize to non-natural domains like digital pathology and heritage, finding that these sparse models outperform larger unpruned counterparts across seven datasets, though generalization is more limited in biomedical contexts.
We study the generalization properties of pruned neural networks that are the winners of the lottery ticket hypothesis on datasets of natural images. We analyse their potential under conditions in which training data is scarce and comes from a non-natural domain. Specifically, we investigate whether pruned models that are found on the popular CIFAR-10/100 and Fashion-MNIST datasets, generalize to seven different datasets that come from the fields of digital pathology and digital heritage. Our results show that there are significant benefits in transferring and training sparse architectures over larger parametrized models, since in all of our experiments pruned networks, winners of the lottery ticket hypothesis, significantly outperform their larger unpruned counterparts. These results suggest that winning initializations do contain inductive biases that are generic to some extent, although, as reported by our experiments on the biomedical datasets, their generalization properties can be more limiting than what has been so far observed in the literature.