ELSA: Partial Weight Freezing for Overhead-Free Sparse Network Deployment
This provides a practical solution for flexible sparse network deployment, though it is incremental as it builds on existing sparsification techniques.
The paper tackles the problem of deploying deep networks at varying sparsity levels by introducing ELSA, which embeds sparse networks within a dense network, allowing easy extraction at prediction time with no or negligible quality loss compared to independently trained sparse networks.
We present ELSA, a practical solution for creating deep networks that can easily be deployed at different levels of sparsity. The core idea is to embed one or more sparse networks within a single dense network as a proper subset of the weights. At prediction time, any sparse model can be extracted effortlessly simply be zeroing out weights according to a predefined mask. ELSA is simple, powerful and highly flexible. It can use essentially any existing technique for network sparsification and network training. In particular, it does not restrict the loss function, architecture or the optimization technique. Our experiments show that ELSA's advantages of flexible deployment comes with no or just a negligible reduction in prediction quality compared to the standard way of using multiple sparse networks that are trained and stored independently.