Functional Knowledge Transfer with Self-supervised Representation Learning
This work addresses a constraint for practitioners using small-scale datasets by enabling functional knowledge transfer through joint optimization.
The paper tackles the problem of applying self-supervised learning to small datasets by proposing a joint training framework that combines self-supervised and supervised tasks, resulting in consistent performance improvements on classification tasks across three visual datasets.
This work investigates the unexplored usability of self-supervised representation learning in the direction of functional knowledge transfer. In this work, functional knowledge transfer is achieved by joint optimization of self-supervised learning pseudo task and supervised learning task, improving supervised learning task performance. Recent progress in self-supervised learning uses a large volume of data, which becomes a constraint for its applications on small-scale datasets. This work shares a simple yet effective joint training framework that reinforces human-supervised task learning by learning self-supervised representations just-in-time and vice versa. Experiments on three public datasets from different visual domains, Intel Image, CIFAR, and APTOS, reveal a consistent track of performance improvements on classification tasks during joint optimization. Qualitative analysis also supports the robustness of learnt representations. Source code and trained models are available on GitHub.