Adjoined Networks: A Training Paradigm with Applications to Network Compression
This addresses the problem of deploying large models on edge devices for computer vision applications, offering a novel training approach that is incremental over existing knowledge distillation methods.
The paper tackles network compression by introducing Adjoined Networks (AN), a training paradigm that jointly trains a base network and a compressed network with shared parameters, achieving 71.8% top-1 accuracy on ImageNet with only 1.8M parameters and 1.6 GFLOPs using ResNet-50, and further proposes Differentiable Adjoined Networks (DAN) to achieve ResNet-50 level accuracy with 3.8× fewer parameters and 2.2× fewer FLOPs.
Compressing deep neural networks while maintaining accuracy is important when we want to deploy large, powerful models in production and/or edge devices. One common technique used to achieve this goal is knowledge distillation. Typically, the output of a static pre-defined teacher (a large base network) is used as soft labels to train and transfer information to a student (or smaller) network. In this paper, we introduce Adjoined Networks, or AN, a learning paradigm that trains both the original base network and the smaller compressed network together. In our training approach, the parameters of the smaller network are shared across both the base and the compressed networks. Using our training paradigm, we can simultaneously compress (the student network) and regularize (the teacher network) any architecture. In this paper, we focus on popular CNN-based architectures used for computer vision tasks. We conduct an extensive experimental evaluation of our training paradigm on various large-scale datasets. Using ResNet-50 as the base network, AN achieves 71.8% top-1 accuracy with only 1.8M parameters and 1.6 GFLOPs on the ImageNet data-set. We further propose Differentiable Adjoined Networks (DAN), a training paradigm that augments AN by using neural architecture search to jointly learn both the width and the weights for each layer of the smaller network. DAN achieves ResNet-50 level accuracy on ImageNet with $3.8\times$ fewer parameters and $2.2\times$ fewer FLOPs.