LoRMA: Low-Rank Multiplicative Adaptation for LLMs
This work addresses efficiency in adapting LLMs for downstream tasks, offering an incremental improvement over existing low-rank adaptation techniques.
The paper tackles the computational expense of full fine-tuning for Large Language Models by proposing LoRMA, a method that uses low-rank multiplicative adaptations instead of additive updates, achieving competitive performance on various evaluation metrics.
Large Language Models have shown remarkable capabilities in the NLP domain. Their effectiveness can mainly be attributed to their ability to adapt to an array of downstream tasks. However, generally, full fine-tuning is a computationally expensive job. To mitigate this, many techniques have been developed that prime efficiency, a prominent one being Low-Rank Adaptation (LoRA). However, LoRA and its variants employ re-parametrized additive updates. In this paper, we propose Low-Rank Multiplicative Adaptation (LoRMA), which shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations. We tackle challenges such as computational complexity and rank bottleneck of matrix multiplication by effectively re-ordering operations and introducing rank inflation strategies. We conduct extensive experiments to demonstrate the effectiveness of our approach in terms of various evaluation metrics.