Hybrid and Unitary Fine-Tuning of Large Language Models: Methods and Benchmarking under Resource Constraints
It addresses resource constraints for deploying large language models in real-world applications, offering a practical and scalable fine-tuning solution.
This paper tackles the computational bottleneck of fine-tuning large language models by introducing a hybrid parameter-efficient fine-tuning method that dynamically combines BOFT and LoRA-GA, achieving up to 2.1 times faster training and 50% memory reduction while approaching full fine-tuning accuracy on benchmarks like GLUE and GSM8K.
Fine-tuning large language models (LLMs) remains a computational bottleneck due to their scale and memory demands. This paper presents a comprehensive evaluation of parameter-efficient fine-tuning (PEFT) techniques, including LoRA, BOFT, LoRA-GA, and uRNN, and introduces a novel hybrid strategy that dynamically integrates BOFT's orthogonal stability with LoRA-GA's gradient-aligned rapid convergence. By computing per-layer adaptive updates guided by gradient norms, the hybrid method achieves superior convergence efficiency and generalization across diverse tasks. We also explore, for the first time, the adaptation of unitary RNN (uRNN) principles to transformer-based LLMs, enhancing gradient stability through structured unitary constraints. Empirical evaluations on four benchmarks -- GLUE, GSM8K, MT-Bench, and HumanEval -- using models ranging from 7B to 405B parameters demonstrate that our hybrid method consistently outperforms individual PEFT baselines, approaching full fine-tuning accuracy while reducing resource consumption by up to 2.1 times in training time and 50 percent in memory usage. These findings establish the hybrid approach as a practical and scalable fine-tuning solution for real-world deployment of LLMs under resource constraints.