Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets
This addresses the need for more efficient adaptation of large foundation models under tight compute and memory budgets, particularly for extreme parameter-efficient scenarios, though it appears incremental as it builds on existing PEFT methods.
The paper tackled the problem of limited granularity and effectiveness in Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA by proposing Wavelet Fine-Tuning (WaveFT), which learns sparse updates in the wavelet domain to allow precise control of trainable parameters and significantly outperforms LoRA and other PEFT methods in personalized text-to-image generation, especially at low parameter counts.
Efficiently adapting large foundation models is critical, especially with tight compute and memory budgets. Parameter-Efficient Fine-Tuning (PEFT) methods such as LoRA offer limited granularity and effectiveness in few-parameter regimes. We propose Wavelet Fine-Tuning (WaveFT), a novel PEFT method that learns highly sparse updates in the wavelet domain of residual matrices. WaveFT allows precise control of trainable parameters, offering fine-grained capacity adjustment and excelling with remarkably low parameter count, potentially far fewer than LoRA's minimum, ideal for extreme parameter-efficient scenarios. Evaluated on personalized text-to-image generation using Stable Diffusion XL as baseline, WaveFT significantly outperforms LoRA and other PEFT methods, especially at low parameter counts; achieving superior subject fidelity, prompt alignment, and image diversity.