Do deep neural networks utilize the weight space efficiently?
Onur Can Koyun Behçet Uğur Töreyin
Deep learning models like Transformers and Convolutional Neural Networks
(CNNs) have revolutionized various domains, but their parameter-intensive
nature hampers deployment in resource-constrained settings. In this paper, we
introduce a novel concept utilizes column space and row space of weight
matrices, which allows for a substantial reduction in model parameters without
compromising performance. Leveraging this paradigm, we achieve
parameter-efficient deep learning models.. Our approach applies to both
Bottleneck and Attention layers, effectively halving the parameters while
incurring only minor performance degradation. Extensive experiments conducted
on the ImageNet dataset with ViT and ResNet50 demonstrate the effectiveness of
our method, showcasing competitive performance when compared to traditional
models. This approach not only addresses the pressing demand for parameter
efficient deep learning solutions but also holds great promise for practical
deployment in real-world scenarios.