Do deep neural networks utilize the weight space efficiently?
This addresses the pressing demand for parameter-efficient deep learning solutions for deployment in resource-constrained settings, representing a novel method rather than an incremental improvement.
The paper tackles the problem of parameter-intensive deep learning models by introducing a novel concept that utilizes column and row spaces of weight matrices, achieving parameter reductions of approximately 50% with only minor performance degradation on ImageNet with ViT and ResNet50.
Deep learning models like Transformers and Convolutional Neural Networks (CNNs) have revolutionized various domains, but their parameter-intensive nature hampers deployment in resource-constrained settings. In this paper, we introduce a novel concept utilizes column space and row space of weight matrices, which allows for a substantial reduction in model parameters without compromising performance. Leveraging this paradigm, we achieve parameter-efficient deep learning models.. Our approach applies to both Bottleneck and Attention layers, effectively halving the parameters while incurring only minor performance degradation. Extensive experiments conducted on the ImageNet dataset with ViT and ResNet50 demonstrate the effectiveness of our method, showcasing competitive performance when compared to traditional models. This approach not only addresses the pressing demand for parameter efficient deep learning solutions but also holds great promise for practical deployment in real-world scenarios.