TLoRA+: A Low-Rank Parameter-Efficient Fine-Tuning Method for Large Language Models
For practitioners fine-tuning LLMs, TLoRA+ offers a more effective yet efficient alternative to LoRA, enhancing task-specific adaptation.
The paper proposes TLoRA+, a low-rank PEFT method that integrates a TLoRA+ optimizer into LLM weight matrices, achieving improved performance over LoRA on the GLUE benchmark without significant computational overhead.
Fine-tuning large language models (LLMs) aims to adapt pre-trained models to specific tasks using relatively small and domain-specific datasets. Among Parameter-Efficient Fine-Tuning (PEFT) methods, Low-Rank Adaptation (LoRA) stands out by matching the performance of full fine-tuning while avoiding additional inference latency. In this paper, we propose a novel PEFT method that incorporates the TLoRA+ optimizer into the weight matrices of pre-trained models. The proposed approach not only preserves the efficiency of low-rank adaptation but also further enhances performance without significantly increasing computational cost. We conduct experiments on the GLUE benchmark across diverse model architectures. Numerical experiments consistently demonstrate the effectiveness and robustness of our proposed method.