Modularized Morphing of Neural Networks
This work addresses network morphism for neural network design and optimization, but it is incremental as it builds on existing layer-level morphing by extending to modular transformations.
The paper tackles the problem of network morphism at a higher level by enabling a convolutional layer to morph into arbitrary neural network modules, proving that any reasonable module can be derived from a single convolutional layer. Experiments on ResNet with benchmark datasets verified the effectiveness of the proposed solution.
In this work we study the problem of network morphism, an effective learning scheme to morph a well-trained neural network to a new one with the network function completely preserved. Different from existing work where basic morphing types on the layer level were addressed, we target at the central problem of network morphism at a higher level, i.e., how a convolutional layer can be morphed into an arbitrary module of a neural network. To simplify the representation of a network, we abstract a module as a graph with blobs as vertices and convolutional layers as edges, based on which the morphing process is able to be formulated as a graph transformation problem. Two atomic morphing operations are introduced to compose the graphs, based on which modules are classified into two families, i.e., simple morphable modules and complex modules. We present practical morphing solutions for both of these two families, and prove that any reasonable module can be morphed from a single convolutional layer. Extensive experiments have been conducted based on the state-of-the-art ResNet on benchmark datasets, and the effectiveness of the proposed solution has been verified.