Rescaling CNN through Learnable Repetition of Network Parameters
This addresses the problem of inefficient parameter usage in CNNs for deep learning practitioners, offering a method to improve model performance without added computational cost, though it is incremental in nature.
The paper investigates whether performance gains in deeper CNNs stem from more parameters or network size, and introduces a rescaling strategy using learnable weight repetition to boost performance without increasing parameters, achieving comparable results with as low as 6% of the parameters of deeper networks.
Deeper and wider CNNs are known to provide improved performance for deep learning tasks. However, most such networks have poor performance gain per parameter increase. In this paper, we investigate whether the gain observed in deeper models is purely due to the addition of more optimization parameters or whether the physical size of the network as well plays a role. Further, we present a novel rescaling strategy for CNNs based on learnable repetition of its parameters. Based on this strategy, we rescale CNNs without changing their parameter count, and show that learnable sharing of weights itself can provide significant boost in the performance of any given model without changing its parameter count. We show that small base networks when rescaled, can provide performance comparable to deeper networks with as low as 6% of optimization parameters of the deeper one. The relevance of weight sharing is further highlighted through the example of group-equivariant CNNs. We show that the significant improvements obtained with group-equivariant CNNs over the regular CNNs on classification problems are only partly due to the added equivariance property, and part of it comes from the learnable repetition of network weights. For rot-MNIST dataset, we show that up to 40% of the relative gain reported by state-of-the-art methods for rotation equivariance could actually be due to just the learnt repetition of weights.