Unsupervised Representation Learning for Binary Networks by Joint Classifier Learning
This work addresses the challenge of deploying efficient models on edge devices by improving unsupervised learning for binary networks, representing an incremental advancement in domain-specific optimization.
The paper tackles the problem of enabling unsupervised representation learning for binary networks to deploy models on resource-limited edge devices, proposing a method called BURN that outperforms self-supervised baselines and sometimes supervised pretraining across five downstream tasks using seven datasets.
Self-supervised learning is a promising unsupervised learning framework that has achieved success with large floating point networks. But such networks are not readily deployable to edge devices. To accelerate deployment of models with the benefit of unsupervised representation learning to such resource limited devices for various downstream tasks, we propose a self-supervised learning method for binary networks that uses a moving target network. In particular, we propose to jointly train a randomly initialized classifier, attached to a pretrained floating point feature extractor, with a binary network. Additionally, we propose a feature similarity loss, a dynamic loss balancing and modified multi-stage training to further improve the accuracy, and call our method BURN. Our empirical validations over five downstream tasks using seven datasets show that BURN outperforms self-supervised baselines for binary networks and sometimes outperforms supervised pretraining. Code is availabe at https://github.com/naver-ai/burn.