Modularizing Deep Learning via Pairwise Learning With Kernels
This work addresses the need for more efficient and maintainable deep learning workflows, offering a modular approach that reduces label requirements and simplifies implementation, though it is incremental in reinterpreting existing architectures.
The paper tackles the problem of modularizing deep learning by proposing a framework that redefines neural networks as stacked linear models, enabling training without between-module backpropagation and leveraging weak supervision for hidden modules. It achieves 94.88% accuracy on CIFAR-10 with only 10 labeled examples using a ResNet-18 backbone.
By redefining the conventional notions of layers, we present an alternative view on finitely wide, fully trainable deep neural networks as stacked linear models in feature spaces, leading to a kernel machine interpretation. Based on this construction, we then propose a provably optimal modular learning framework for classification that does not require between-module backpropagation. This modular approach brings new insights into the label requirement of deep learning: It leverages only implicit pairwise labels (weak supervision) when learning the hidden modules. When training the output module, on the other hand, it requires full supervision but achieves high label efficiency, needing as few as 10 randomly selected labeled examples (one from each class) to achieve 94.88% accuracy on CIFAR-10 using a ResNet-18 backbone. Moreover, modular training enables fully modularized deep learning workflows, which then simplify the design and implementation of pipelines and improve the maintainability and reusability of models. To showcase the advantages of such a modularized workflow, we describe a simple yet reliable method for estimating reusability of pre-trained modules as well as task transferability in a transfer learning setting. At practically no computation overhead, it precisely described the task space structure of 15 binary classification tasks from CIFAR-10.