Network with Sub-Networks
This is an incremental improvement for neural network architecture design, potentially benefiting model deployment and flexibility.
The paper tackles the problem of enabling neural networks to have detachable weight layers for sub-networks during inference, achieving test accuracy comparable to regularly trained models while maintaining this detachability.
We introduce network with sub-networks, a neural network which its weight layers could be detached into sub-neural networks during inference. To develop weights and biases which could be inserted in both base and sub-neural networks, firstly, the parameters are copied from sub-model to base-model. Each model is forward-propagated separately. Gradients from a pair of networks are averaged and, used to update both networks. Our base model achieves the test-accuracy which is comparable to the regularly trained models, while the model maintains the ability to detach weight layers.