Network of Theseus (like the ship)
This work addresses the problem of limited architectural flexibility in deep learning for researchers and practitioners, allowing for better accuracy-efficiency tradeoffs and design exploration, though it is incremental as it builds on existing representational similarity techniques.
The paper tackles the constraint that neural network architectures must remain fixed from training to deployment by introducing Network of Theseus (NoT), a method that progressively converts a guide network into a different target architecture while preserving performance, enabling conversions like convolutional networks to multilayer perceptrons or GPT-2 to recurrent neural networks.
A standard assumption in deep learning is that the inductive bias introduced by a neural network architecture must persist from training through inference. The architecture you train with is the architecture you deploy. This assumption constrains the community from selecting architectures that may have desirable efficiency or design properties due to difficulties with optimization. We challenge this assumption with Network of Theseus (NoT), a method for progressively converting a trained, or even untrained, guide network architecture part-by-part into an entirely different target network architecture while preserving the performance of the guide network. At each stage, components in the guide network architecture are incrementally replaced with target architecture modules and aligned via representational similarity metrics. This procedure largely preserves the functionality of the guide network even under substantial architectural changes-for example, converting a convolutional network into a multilayer perceptron, or GPT-2 into a recurrent neural network. By decoupling optimization from deployment, NoT expands the space of viable inference-time architectures, opening opportunities for better accuracy-efficiency tradeoffs and enabling more directed exploration of the architectural design space.