Scholar Feed

Parameter-efficient fine-tuning (LoRA family)

Adapter

Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning

Heavily superseded#6 of 1,113 most-superseded · first seen Nov 18, 2023

Heavily superseded — a standard baseline that newer methods routinely beat

10 papers critique it · 23 beat it on benchmarks

What papers say

Verbatim critique sentences, each from a paper that cites Adapter as a baseline.

While various strategies are used to decrease trainable parameters, our method of sharing adapters across blocks offers a distinct advantage on efficient adaptation on various tasks.
— Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision
“parameter-efficient fine-tuning techniques, such as adapters, are less effective as they can result in catastrophic forgetting of existing languages”
— Dual-Pipeline with Low-Rank Adaptation for New Language Integration in Multilingual ASR
“These methods incur additional inference overhead due to computing the inserted modules.”
— AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning
“In the Adapter approach, only the bottleneck-like MLP modules are updated, while all other parameters, including the self-attention modules, remain frozen.”
— Enhancing Parameter-Efficient Fine-Tuning of Vision Transformers through Frequency-Based Adaptation
“While effective, these methods introduce inference overhead.”
— Task-Aware Parameter-Efficient Fine-Tuning of Large Pre-Trained Models at the Edge
“However, this approach overlooks the potential benefits of pre-training for these new parameters.”
— Pre-training Everywhere: Parameter-Efficient Fine-Tuning for Medical Image Analysis via Target Parameter Pre-training
“Compared to Adapter and VPT, we find that VPT benefits more for DML tasks and outperforms the full fine-tuning method (when combined with BitFit).”
— Learning Semantic Proxies from Visual Prompts for Parameter-Efficient Fine-Tuning in Deep Metric Learning
“Although AdapterL has a higher number of parameters compared to LoRA, it does not result in higher performance. This indicates that the location within the architecture, where PEFT modules are applied, does influence the results.”
— Parameter-Efficient Fine-Tuning of Vision Foundation Model for Forest Floor Segmentation from UAV Imagery
“PEFT methods with low-rank bottlenecks, such as Adapter and LoRA, are currently not designed for down-stream tasks involving another modality, such as vision-language (VL) tasks.”
— Introducing Routing Functions to Vision-Language Parameter-Efficient Fine-Tuning with Low-Rank Bottlenecks
“It can be noticed that adapter adds task dependent parameters and incurs inference delay.”
— Parameter-Efficient Fine-Tuning with Circulant and Diagonal Vectors

Beaten on benchmarks

Head-to-head results where a newer method reports beating Adapter. Values are copied from the source paper's tables — verify against the cited paper.

What to use instead

Recent methods in the same sub-problem, not yet superseded in the knowledge base — arXiv benchmark leaders, not vetted production recommendations.