StitchNet: Composing Neural Networks from Pre-Trained Fragments
This addresses the challenge of high compute and data needs in neural network creation, offering a more efficient alternative for model development, though it is incremental in building on existing pre-trained models.
The paper tackles the problem of creating neural networks without extensive training by proposing StitchNet, which stitches together pre-trained fragments using Centered Kernel Alignment for compatibility, achieving accuracy comparable to traditionally trained networks while reducing compute and data requirements.
We propose StitchNet, a novel neural network creation paradigm that stitches together fragments (one or more consecutive network layers) from multiple pre-trained neural networks. StitchNet allows the creation of high-performing neural networks without the large compute and data requirements needed under traditional model creation processes via backpropagation training. We leverage Centered Kernel Alignment (CKA) as a compatibility measure to efficiently guide the selection of these fragments in composing a network for a given task tailored to specific accuracy needs and computing resource constraints. We then show that these fragments can be stitched together to create neural networks with accuracy comparable to that of traditionally trained networks at a fraction of computing resource and data requirements. Finally, we explore a novel on-the-fly personalized model creation and inference application enabled by this new paradigm. The code is available at https://github.com/steerapi/stitchnet.