LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization
This work addresses the problem of inefficient fine-tuning in domain specialization tasks for machine learning practitioners, offering an incremental improvement over existing PEFT methods.
The paper tackles the computational inefficiency and sub-optimal performance of existing Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA by proposing LoSiA, which dynamically localizes and optimizes critical parameters via gradient sparsity analysis, achieving minimal performance drop compared to full fine-tuning with reduced training time, including a faster implementation (LoSiA-Pro) that cuts training latency by about 27% compared to LoRA.
Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA, significantly reduce the number of trainable parameters by introducing low-rank decomposition matrices. However, existing methods perform extensive matrix multiplications in domain specialization tasks, resulting in computational inefficiency and sub-optimal fine-tuning performance. Hence, we propose LoSiA(Low-Resources Subnet Integration Adaptation), an innovative method that dynamically localizes and optimizes critical parameters during the training process. Specifically, it identifies a sub-network using gradient sparsity analysis and optimizes it as the trainable target. This design enables effective high-rank adaptation by updating only the sub-network parameters, reducing the additional matrix multiplication. We also present LoSiA-Pro, a faster implementation of LoSiA, which reduces the training latency by about $27\%$ compared to LoRA. Extensive evaluations show that our method achieves minimal performance drop compared to full fine-tuning, while requiring the least training time across domain specialization and common-sense reasoning tasks. Further analysis shows that LoSiA also reduces forgetting during continued training. The source code is available at https://github.com/KlozeWang/LoSiA.