MoPEFT: A Mixture-of-PEFTs for the Segment Anything Model
This addresses the challenge of efficiently fine-tuning large foundation models like SAM for specific applications, though it is incremental as it builds on existing PEFT and Mixture-of-Experts methods.
The paper tackles the problem of selecting the most appropriate Parameter-Efficient Fine-Tuning (PEFT) method for adapting the Segment Anything Model (SAM) to new domains by proposing MoPEFT, a framework that dynamically combines three PEFT techniques, and shows it consistently outperforms other methods on the MESS benchmark.
The emergence of foundation models, such as the Segment Anything Model (SAM), has sparked interest in Parameter-Efficient Fine-Tuning (PEFT) methods that tailor these large models to application domains outside their training data. However, different PEFT techniques modify the representation of a model differently, making it a non-trivial task to select the most appropriate method for the domain of interest. We propose a new framework, Mixture-of-PEFTs methods (MoPEFT), that is inspired by traditional Mixture-of-Experts (MoE) methodologies and is utilized for fine-tuning SAM. Our MoPEFT framework incorporates three different PEFT techniques as submodules and dynamically learns to activate the ones that are best suited for a given data-task setup. We test our method on the Segment Anything Model and show that MoPEFT consistently outperforms other fine-tuning methods on the MESS benchmark.