TensLoRA: Tensor Alternatives for Low-Rank Adaptation
This work addresses the need for more efficient and flexible adaptation methods in machine learning, particularly for vision and language tasks, though it is incremental as it builds on existing tensor-based approaches.
The paper tackled the problem of inefficient independent low-rank adaptations in Transformers by introducing TensLoRA, a unified framework that aggregates updates into higher-order tensors, enabling mode-specific compression and sometimes outperforming standard LoRA under similar parameter counts.
Low-Rank Adaptation (LoRA) is widely used to efficiently adapt Transformers by adding trainable low-rank matrices to attention projections. While effective, these matrices are considered independent for each attention projection (Query, Key, and Value) and each layer. Recent extensions have considered joint, tensor-based adaptations, but only in limited forms and without a systematic framework. We introduce TensLoRA, a unified framework that aggregates LoRA updates into higher-order tensors and models a broad family of tensor-based low-rank adaptations. Our formulation generalizes existing tensor-based methods and enables mode-specific compression rates, allowing parameter budgets to be tailored according to the modality and task. Experiments on vision and language benchmarks reveal that the tensor construction directly impacts performance, sometimes better than standard LoRA under similar parameter counts.