FPS: Feedforward-based Parameter Selection For Efficient Fine-Tuning
This provides a memory-efficient and practical solution for fine-tuning large-scale pre-trained models, addressing a key bottleneck in parameter-efficient fine-tuning for researchers and practitioners.
The paper tackled the problem of inefficient parameter selection in fine-tuning large pre-trained models by proposing FPS, a gradient-free method that selects parameters in a single forward pass, achieving comparable performance to state-of-the-art methods while reducing peak memory usage by nearly 9× and accelerating selection by about 2× on 24 visual tasks.
Parameter-Efficient Fine-Tuning (PEFT) has emerged as a key strategy for adapting large-scale pre-trained models to downstream tasks, but existing approaches face notable limitations. Addition-based methods, such as Adapters [1], introduce inference latency and engineering complexity, while selection-based methods like Gradient-based Parameter Selection (GPS) [2] require a full backward pass, which results in the same peak memory usage as full fine-tuning. To address this dilemma, we propose Feedforward-based Parameter Selection (FPS), a gradient-free method that identifies an optimal parameter subset in a single forward pass. FPS ranks parameters by the product of their magnitudes and corresponding input activations, leveraging both pre-trained knowledge and downstream data. Evaluated on $24$ visual tasks from FGVC and VTAB-1k, FPS achieves performance comparable to state-of-the-art methods while reducing peak memory usage by nearly $9 \times$ and accelerating parameter selection by about $2 \times$, offering a genuinely memory-efficient and practical solution for fine-tuning large-scale pre-trained models.