How to Train Your Metamorphic Deep Neural Network
This work provides a scalable solution for adaptable and efficient deployment of deep models, though it is incremental as it builds on the existing NeuMeta paradigm.
The paper tackles the limitation of Neural Metamorphosis (NeuMeta) in generating compressed neural networks, proposing a training algorithm that enables full-network metamorphosis with minimal accuracy degradation, achieving competitive accuracy across various compression ratios.
Neural Metamorphosis (NeuMeta) is a recent paradigm for generating neural networks of varying width and depth. Based on Implicit Neural Representation (INR), NeuMeta learns a continuous weight manifold, enabling the direct generation of compressed models, including those with configurations not seen during training. While promising, the original formulation of NeuMeta proves effective only for the final layers of the undelying model, limiting its broader applicability. In this work, we propose a training algorithm that extends the capabilities of NeuMeta to enable full-network metamorphosis with minimal accuracy degradation. Our approach follows a structured recipe comprising block-wise incremental training, INR initialization, and strategies for replacing batch normalization. The resulting metamorphic networks maintain competitive accuracy across a wide range of compression ratios, offering a scalable solution for adaptable and efficient deployment of deep models. The code is available at: https://github.com/TSommariva/HTTY_NeuMeta.