Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptation
This work addresses parameter-efficient fine-tuning for large-scale neural models, offering a novel method that enhances performance without increasing trainable parameters, though it appears incremental as it builds on existing LoRA and tensor-train approaches.
The paper tackled the limited expressivity and generalization of standard Low-Rank Adaptation (LoRA) by proposing TensorGuide, a tensor-train-guided framework that generates correlated low-rank matrices, resulting in improved accuracy and scalability with fewer parameters on benchmarks like quantum dot classification and GPT-2 fine-tuning.
Low-Rank Adaptation (LoRA) is widely recognized for its parameter-efficient fine-tuning of large-scale neural models. However, standard LoRA independently optimizes low-rank matrices, which inherently limits its expressivity and generalization capabilities. While classical tensor-train (TT) decomposition can be separately employed on individual LoRA matrices, this work demonstrates that the classical TT-based approach neither significantly improves parameter efficiency nor achieves substantial performance gains. This paper proposes TensorGuide, a novel tensor-train-guided adaptation framework to overcome these limitations. TensorGuide generates two correlated low-rank LoRA matrices through a unified TT structure driven by controlled Gaussian noise. The resulting joint TT representation inherently provides structured, low-rank adaptations, significantly enhancing expressivity, generalization, and parameter efficiency without increasing the number of trainable parameters. Theoretically, we justify these improvements through neural tangent kernel analyses, demonstrating superior optimization dynamics and enhanced generalization. Extensive experiments on quantum dot classification and GPT-2 fine-tuning benchmarks demonstrate that TensorGuide-based LoRA consistently outperforms standard LoRA and TT-LoRA, achieving improved accuracy and scalability with fewer parameters.