Revealing the Utilized Rank of Subspaces of Learning in Neural Networks
This work addresses the problem of inefficient parameter usage in neural networks for researchers and practitioners, offering insights into model compression and efficiency, though it is incremental in nature.
The authors tackled the problem of understanding how well neural network weights utilize the available space, revealing that most models use only a fraction of it, such as 35% for ViTB-16 and 20% for ViTL-16 on ImageNet, and their transformation reduces parameters by 50% and 25% with minimal accuracy loss.
In this work, we study how well the learned weights of a neural network utilize the space available to them. This notion is related to capacity, but additionally incorporates the interaction of the network architecture with the dataset. Most learned weights appear to be full rank, and are therefore not amenable to low rank decomposition. This deceptively implies that the weights are utilizing the entire space available to them. We propose a simple data-driven transformation that projects the weights onto the subspace where the data and the weight interact. This preserves the functional mapping of the layer and reveals its low rank structure. In our findings, we conclude that most models utilize a fraction of the available space. For instance, for ViTB-16 and ViTL-16 trained on ImageNet, the mean layer utilization is 35% and 20% respectively. Our transformation results in reducing the parameters to 50% and 25% respectively, while resulting in less than 0.2% accuracy drop after fine-tuning. We also show that self-supervised pre-training drives this utilization up to 70%, justifying its suitability for downstream tasks.