On the Transferability of Representations in Neural Networks Between Datasets and Tasks
This study addresses the problem of understanding representation transferability for researchers in machine learning, but it appears incremental as it builds on existing paradigms like transfer learning.
The paper investigates how representations in deep neural networks transfer across different datasets and tasks, noting empirical observations on layer-wise transferability.
Deep networks, composed of multiple layers of hierarchical distributed representations, tend to learn low-level features in initial layers and transition to high-level features towards final layers. Paradigms such as transfer learning, multi-task learning, and continual learning leverage this notion of generic hierarchical distributed representations to share knowledge across datasets and tasks. Herein, we study the layer-wise transferability of representations in deep networks across a few datasets and tasks and note some interesting empirical observations.