GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Models
This addresses the need for more efficient adaptation of large models for downstream tasks, offering a novel approach that improves over existing PEFT methods, though it appears incremental as it builds on prior PEFT techniques.
The paper tackles the problem of inefficient training of task-specific parameter updates in Parameter-Efficient Fine-Tuning (PEFT) for Pretrained Foundation Models by proposing GenFT, which extracts structured information from pretrained weights to guide updates, achieving superior performance on CV and NLP benchmarks like VTAB-1K, FGVC, and GLUE.
Pretrained Foundation Models (PFMs) have transformed numerous applications by enabling efficient adaptation to customized tasks. Parameter-Efficient Fine-Tuning (PEFT) has emerged as a resource-efficient alternative to full fine-tuning, especially leveraging reparameterized weights $ΔW$ to adapt models for downstream tasks. However, a critical yet underexplored question remains: can we utilize well-pretrained weights $W_0$ to guide the update of task-specific $ΔW$, avoiding inefficient training it from scratch? To end this, we propose Generative Parameter-Efficient Fine-Tuning (GenFT), a novel method that extracts structured, transferable information from $W_0$ for efficient $ΔW$ training. To extract row and column structure information, GenFT applies row and column transformations to distill essential patterns from $W_0$. A tailored policy further decomposes $ΔW$ into layer-shared and layer-specific components, balancing information reuse and individualized flexibility. GenFT is simple yet effective, achieving superior performance across CV and NLP tasks. Extensive experiments on VTAB-1K, FGVC, and GLUE benchmarks demonstrate that GenFT outperforms state-of-the-art PEFT methods, offering a new perspective for efficient model adaptation.