Block Circulant Adapter for Large Language Models
This addresses the problem of expensive fine-tuning for users of large language models, offering a more efficient approach with incremental improvements over existing methods.
The paper tackles the high cost of fine-tuning large language models by proposing a block circulant matrix-based method that reduces storage and computation, achieving up to 32x fewer FLOPs and 14x fewer parameters while maintaining competitive task performance.
Fine-tuning large language models (LLMs) is difficult due to their huge model size. Recent Fourier domain-based methods show potential for reducing fine-tuning costs. We propose a block circulant matrix-based fine-tuning method with a stable training heuristic to leverage the properties of circulant matrices and one-dimensional Fourier transforms to reduce storage and computation costs. Experiments show that our method uses $14\times$ less number of parameters than VeRA, $16\times$ smaller than LoRA and $32\times$ less FLOPs than FourierFT, while maintaining close or better task performance. Our approach presents a promising way in frequency domain to fine-tune large models on downstream tasks.