Towards a More Complete Theory of Function Preserving Transforms
This work addresses the need for more flexible and efficient architecture search in deep learning, though it is incremental as it builds on existing function preserving transforms like Net2Net and Network Morphisms.
The paper tackles the problem of altering neural network architectures while preserving their functions, introducing R2R methods that integrate residual connections into function preserving transforms, resulting in competitive performance and faster training with more diverse filters in image classification tasks.
In this paper, we develop novel techniques that can be used to alter the architecture of a neural network, while maintaining the function it represents. Such operations are known as function preserving transforms and have proven useful in transferring knowledge between networks to evaluate architectures quickly, thus having applications in efficient architectures searches. Our methods allow the integration of residual connections into function preserving transforms, so we call them R2R. We provide a derivation for R2R and show that it yields competitive performance with other function preserving transforms, thereby decreasing the restrictions on deep learning architectures that can be extended through function preserving transforms. We perform a comparative analysis with other function preserving transforms such as Net2Net and Network Morphisms, where we shed light on their differences and individual use cases. Finally, we show the effectiveness of R2R to train models quickly, as well as its ability to learn a more diverse set of filters on image classification tasks compared to Net2Net and Network Morphisms.