Efficient Learning With Sine-Activated Low-rank Matrices
This addresses the problem of maintaining accuracy while reducing parameters in neural networks for researchers and practitioners in machine learning, though it appears incremental as a plug-in enhancement to existing low-rank methods.
The paper tackles the trade-off between parameter efficiency and accuracy in low-rank neural network decompositions by integrating a sinusoidal function to increase rank and enhance performance, achieving improved results across Vision Transformers, Large Language Models, Neural Radiance Fields, and 3D shape modeling.
Low-rank decomposition has emerged as a vital tool for enhancing parameter efficiency in neural network architectures, gaining traction across diverse applications in machine learning. These techniques significantly lower the number of parameters, striking a balance between compactness and performance. However, a common challenge has been the compromise between parameter efficiency and the accuracy of the model, where reduced parameters often lead to diminished accuracy compared to their full-rank counterparts. In this work, we propose a novel theoretical framework that integrates a sinusoidal function within the low-rank decomposition process. This approach not only preserves the benefits of the parameter efficiency characteristic of low-rank methods but also increases the decomposition's rank, thereby enhancing model performance. Our method proves to be a plug in enhancement for existing low-rank models, as evidenced by its successful application in Vision Transformers (ViT), Large Language Models (LLMs), Neural Radiance Fields (NeRF) and 3D shape modelling.